[==========] Running 21 tests from 1 test case. [----------] Global test environment set-up. [----------] 21 tests from RemoteKsckTest [ RUN ] RemoteKsckTest.TestClusterOk WARNING: Logging before InitGoogleLogging() is written to STDERR I0918 20:38:09.997192 19320 test_util.cc:212] Using random seed: -794082038 I0918 20:38:10.000159 19320 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.18.222.62:44223,127.18.222.61:43611,127.18.222.60:39395 I0918 20:38:10.017616 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 122592us I0918 20:38:10.018223 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:10.018383 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:10.018707 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:10.018865 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:10.022825 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/master-0-root/instance: uuid: "a9903afeb95a4ed8b8110650f809f965" format_stamp: "Formatted at 2019-09-18 20:38:10 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:10.036556 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.013s user 0.015s sys 0.001s I0918 20:38:10.037885 19320 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0918 20:38:10.038540 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:10.047238 19320 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s I0918 20:38:10.047418 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/master-0-root uuid: "a9903afeb95a4ed8b8110650f809f965" format_stamp: "Formatted at 2019-09-18 20:38:10 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:10.047770 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0918 20:38:10.787902 19320 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.709s user 0.331s sys 0.355s W0918 20:38:10.788154 19320 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.709s user 0.331s sys 0.355s I0918 20:38:10.855482 19320 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0918 20:38:10.910859 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:44223 I0918 20:38:10.913528 19320 webserver.cc:374] Webserver started at http://127.18.222.62:39733/ using document root and password file I0918 20:38:10.916674 19381 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:10.942768 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 122592us I0918 20:38:10.943158 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:10.943323 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:10.943562 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:10.943739 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:10.944708 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/master-1-root/instance: uuid: "200998a474bb42b2a13b5ec292154c78" format_stamp: "Formatted at 2019-09-18 20:38:10 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:10.944597 19381 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:11.025579 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.080s user 0.004s sys 0.008s I0918 20:38:11.027153 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:11.040508 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0918 20:38:11.040700 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/master-1-root uuid: "200998a474bb42b2a13b5ec292154c78" format_stamp: "Formatted at 2019-09-18 20:38:10 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:11.040998 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:11.054800 19381 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } has no permanent_uuid. Determining permanent_uuid... W0918 20:38:11.056946 19381 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:43611: Network error: Client connection negotiation failed: client connection to 127.18.222.61:43611: connect: Connection refused (error 111) I0918 20:38:11.096521 19381 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } attempt: 1 W0918 20:38:11.098562 19381 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:43611: Network error: Client connection negotiation failed: client connection to 127.18.222.61:43611: connect: Connection refused (error 111) I0918 20:38:11.158203 19381 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } attempt: 2 I0918 20:38:11.160212 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:43611 I0918 20:38:11.163224 19320 webserver.cc:374] Webserver started at http://127.18.222.61:46425/ using document root and password file I0918 20:38:11.166577 19439 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:11.172119 19439 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:11.177335 19381 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } has no permanent_uuid. Determining permanent_uuid... W0918 20:38:11.179852 19381 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:39395: Network error: Client connection negotiation failed: client connection to 127.18.222.60:39395: connect: Connection refused (error 111) I0918 20:38:11.187350 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 123092us I0918 20:38:11.187898 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:11.188108 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:11.188411 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:11.188530 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:11.189851 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/master-2-root/instance: uuid: "db6ca21578fe4f5a86e7d466226a447a" format_stamp: "Formatted at 2019-09-18 20:38:11 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:11.191146 19439 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:11.201118 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.012s sys 0.000s I0918 20:38:11.201161 19439 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:11.202999 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 W0918 20:38:11.203088 19439 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:39395: Network error: Client connection negotiation failed: client connection to 127.18.222.60:39395: connect: Connection refused (error 111) I0918 20:38:11.211422 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:38:11.211630 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/master-2-root uuid: "db6ca21578fe4f5a86e7d466226a447a" format_stamp: "Formatted at 2019-09-18 20:38:11 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:11.211962 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:11.236573 19381 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } attempt: 1 W0918 20:38:11.238629 19381 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:39395: Network error: Client connection negotiation failed: client connection to 127.18.222.60:39395: connect: Connection refused (error 111) I0918 20:38:11.257633 19439 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } attempt: 1 W0918 20:38:11.260080 19439 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:39395: Network error: Client connection negotiation failed: client connection to 127.18.222.60:39395: connect: Connection refused (error 111) I0918 20:38:11.271224 19381 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } attempt: 2 W0918 20:38:11.274541 19381 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:39395: Network error: Client connection negotiation failed: client connection to 127.18.222.60:39395: connect: Connection refused (error 111) I0918 20:38:11.293088 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:39395 I0918 20:38:11.295253 19320 webserver.cc:374] Webserver started at http://127.18.222.60:45303/ using document root and password file I0918 20:38:11.297811 19495 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:11.299571 19439 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } attempt: 2 I0918 20:38:11.299813 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0918 20:38:11.302893 19495 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:11.320991 19439 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78: Bootstrap starting. I0918 20:38:11.322037 19495 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:11.325796 19439 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78: No blocks or log segments found. Creating new log. I0918 20:38:11.328073 19439 log.cc:525] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78: Log is configured to *not* fsync() on all Append() calls I0918 20:38:11.331966 19439 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78: No bootstrap required, opened a new log I0918 20:38:11.333320 19495 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:11.348387 19495 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a: Bootstrap starting. I0918 20:38:11.348855 19439 raft_consensus.cc:340] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } I0918 20:38:11.349475 19439 raft_consensus.cc:366] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:11.349702 19439 raft_consensus.cc:705] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 200998a474bb42b2a13b5ec292154c78, State: Initialized, Role: FOLLOWER I0918 20:38:11.350447 19439 consensus_queue.cc:229] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } I0918 20:38:11.351322 19495 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a: No blocks or log segments found. Creating new log. I0918 20:38:11.352820 19439 sys_catalog.cc:340] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } } I0918 20:38:11.353438 19439 sys_catalog.cc:343] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:11.353271 19499 sys_catalog.cc:340] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } } I0918 20:38:11.354230 19499 sys_catalog.cc:343] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:11.355074 19439 sys_catalog.cc:428] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:38:11.357296 19495 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a: No bootstrap required, opened a new log W0918 20:38:11.358203 19394 tablet.cc:1763] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78: Can't schedule compaction. Clean time has not been advanced past its initial value. I0918 20:38:11.359593 19495 raft_consensus.cc:340] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } I0918 20:38:11.360357 19495 raft_consensus.cc:366] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:11.360585 19495 raft_consensus.cc:705] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6ca21578fe4f5a86e7d466226a447a, State: Initialized, Role: FOLLOWER I0918 20:38:11.361186 19495 consensus_queue.cc:229] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } W0918 20:38:11.363659 19501 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:38:11.364006 19501 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:11.364219 19495 sys_catalog.cc:340] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } } I0918 20:38:11.364459 19503 sys_catalog.cc:340] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } } I0918 20:38:11.365034 19495 sys_catalog.cc:343] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:11.365097 19503 sys_catalog.cc:343] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:11.366231 19495 sys_catalog.cc:428] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a [sys.catalog]: configured and running, proceeding with master startup. W0918 20:38:11.370152 19504 catalog_manager.cc:1155] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:38:11.370371 19504 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:11.371937 19381 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } attempt: 3 I0918 20:38:11.387926 19381 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965: Bootstrap starting. I0918 20:38:11.391902 19381 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965: No blocks or log segments found. Creating new log. I0918 20:38:11.396724 19381 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965: No bootstrap required, opened a new log I0918 20:38:11.398866 19381 raft_consensus.cc:340] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } I0918 20:38:11.399631 19381 raft_consensus.cc:366] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:11.399914 19381 raft_consensus.cc:705] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9903afeb95a4ed8b8110650f809f965, State: Initialized, Role: FOLLOWER I0918 20:38:11.400574 19381 consensus_queue.cc:229] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } I0918 20:38:11.403131 19381 sys_catalog.cc:340] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } } I0918 20:38:11.403661 19381 sys_catalog.cc:343] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:11.403548 19506 sys_catalog.cc:340] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } } I0918 20:38:11.404547 19506 sys_catalog.cc:343] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:11.404669 19381 sys_catalog.cc:428] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:38:11.406265 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 I0918 20:38:11.406489 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 W0918 20:38:11.409523 19507 catalog_manager.cc:1155] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:38:11.409795 19507 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:11.423938 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 123092us I0918 20:38:11.424335 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:11.424489 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:11.424736 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:11.424860 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:11.425915 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/ts-0-root/instance: uuid: "37f57f2c32fe437fa6df8bf99c0a7610" format_stamp: "Formatted at 2019-09-18 20:38:11 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:11.437459 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.013s sys 0.000s I0918 20:38:11.438956 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:11.448216 19320 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s I0918 20:38:11.448406 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/ts-0-root uuid: "37f57f2c32fe437fa6df8bf99c0a7610" format_stamp: "Formatted at 2019-09-18 20:38:11 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:11.448696 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:11.457662 19499 raft_consensus.cc:472] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:11.458303 19499 raft_consensus.cc:494] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } I0918 20:38:11.462350 19358 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "200998a474bb42b2a13b5ec292154c78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9903afeb95a4ed8b8110650f809f965" is_pre_election: true I0918 20:38:11.462533 19472 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "200998a474bb42b2a13b5ec292154c78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6ca21578fe4f5a86e7d466226a447a" is_pre_election: true I0918 20:38:11.463282 19358 raft_consensus.cc:2350] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 200998a474bb42b2a13b5ec292154c78 in term 0. I0918 20:38:11.463454 19472 raft_consensus.cc:2350] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 200998a474bb42b2a13b5ec292154c78 in term 0. I0918 20:38:11.462873 19499 leader_election.cc:296] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9903afeb95a4ed8b8110650f809f965 (127.18.222.62:44223), db6ca21578fe4f5a86e7d466226a447a (127.18.222.60:39395) I0918 20:38:11.464915 19387 leader_election.cc:310] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 200998a474bb42b2a13b5ec292154c78, a9903afeb95a4ed8b8110650f809f965; no voters: I0918 20:38:11.465859 19499 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:11.466167 19499 raft_consensus.cc:472] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:11.466490 19499 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:11.472484 19499 raft_consensus.cc:494] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } I0918 20:38:11.475137 19358 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "200998a474bb42b2a13b5ec292154c78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9903afeb95a4ed8b8110650f809f965" I0918 20:38:11.475214 19472 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "200998a474bb42b2a13b5ec292154c78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6ca21578fe4f5a86e7d466226a447a" I0918 20:38:11.475647 19358 raft_consensus.cc:2890] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:11.475795 19472 raft_consensus.cc:2890] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:11.476339 19499 leader_election.cc:296] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [CANDIDATE]: Term 1 election: Requested vote from peers a9903afeb95a4ed8b8110650f809f965 (127.18.222.62:44223), db6ca21578fe4f5a86e7d466226a447a (127.18.222.60:39395) I0918 20:38:11.485734 19472 raft_consensus.cc:2350] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 200998a474bb42b2a13b5ec292154c78 in term 1. I0918 20:38:11.486915 19358 raft_consensus.cc:2350] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 200998a474bb42b2a13b5ec292154c78 in term 1. I0918 20:38:11.487260 19389 leader_election.cc:310] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 200998a474bb42b2a13b5ec292154c78, db6ca21578fe4f5a86e7d466226a447a; no voters: I0918 20:38:11.488724 19499 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:11.492314 19499 raft_consensus.cc:667] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [term 1 LEADER]: Becoming Leader. State: Replica: 200998a474bb42b2a13b5ec292154c78, State: Running, Role: LEADER I0918 20:38:11.493120 19499 consensus_queue.cc:206] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } I0918 20:38:11.501605 19514 sys_catalog.cc:340] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 200998a474bb42b2a13b5ec292154c78. Latest consensus state: current_term: 1 leader_uuid: "200998a474bb42b2a13b5ec292154c78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } } I0918 20:38:11.502562 19514 sys_catalog.cc:343] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [sys.catalog]: This master's current role is: LEADER I0918 20:38:11.509137 19516 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:38:11.514138 19516 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:38:11.549770 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:11.550163 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:11.552562 19472 raft_consensus.cc:1184] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a [term 1 FOLLOWER]: Refusing update from remote peer 200998a474bb42b2a13b5ec292154c78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:11.553503 19358 raft_consensus.cc:1184] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965 [term 1 FOLLOWER]: Refusing update from remote peer 200998a474bb42b2a13b5ec292154c78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:11.554976 19514 consensus_queue.cc:984] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:38:11.555891 19499 consensus_queue.cc:984] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:11.567780 19506 sys_catalog.cc:340] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 200998a474bb42b2a13b5ec292154c78. Latest consensus state: current_term: 1 leader_uuid: "200998a474bb42b2a13b5ec292154c78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } } I0918 20:38:11.568642 19506 sys_catalog.cc:343] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:11.572199 19503 sys_catalog.cc:340] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 200998a474bb42b2a13b5ec292154c78. Latest consensus state: current_term: 1 leader_uuid: "200998a474bb42b2a13b5ec292154c78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } } I0918 20:38:11.573091 19503 sys_catalog.cc:343] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:11.575709 19520 mvcc.cc:203] Tried to move safe_time back from 6425964918978719744 to 6425964918764036096. Current Snapshot: MvccSnapshot[committed={T|T < 6425964918978719744}] I0918 20:38:11.584007 19514 sys_catalog.cc:340] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "200998a474bb42b2a13b5ec292154c78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } } I0918 20:38:11.585392 19514 sys_catalog.cc:343] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [sys.catalog]: This master's current role is: LEADER I0918 20:38:11.587779 19516 catalog_manager.cc:981] Generated new certificate authority record I0918 20:38:11.591169 19514 sys_catalog.cc:340] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "200998a474bb42b2a13b5ec292154c78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } } I0918 20:38:11.593078 19514 sys_catalog.cc:343] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [sys.catalog]: This master's current role is: LEADER I0918 20:38:11.598392 19516 catalog_manager.cc:1106] Loading token signing keys... I0918 20:38:11.602579 19506 sys_catalog.cc:340] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "200998a474bb42b2a13b5ec292154c78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } } I0918 20:38:11.603394 19506 sys_catalog.cc:343] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:11.612347 19503 sys_catalog.cc:340] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "200998a474bb42b2a13b5ec292154c78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9903afeb95a4ed8b8110650f809f965" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44223 } } peers { permanent_uuid: "200998a474bb42b2a13b5ec292154c78" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43611 } } peers { permanent_uuid: "db6ca21578fe4f5a86e7d466226a447a" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39395 } } } I0918 20:38:11.613342 19503 sys_catalog.cc:343] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:11.627826 19516 catalog_manager.cc:4446] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78: Generated new TSK 0 I0918 20:38:11.636095 19516 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:38:11.736182 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:33889 I0918 20:38:11.738828 19320 webserver.cc:374] Webserver started at http://127.18.222.1:38605/ using document root and password file I0918 20:38:11.775967 19585 heartbeater.cc:322] Connected to a master server at 127.18.222.61:43611 I0918 20:38:11.776413 19585 heartbeater.cc:406] Registering TS with master... I0918 20:38:11.777390 19585 heartbeater.cc:455] Master 127.18.222.61:43611 requested a full tablet report, sending... I0918 20:38:11.782140 19405 ts_manager.cc:186] Registered new tserver with Master: 37f57f2c32fe437fa6df8bf99c0a7610 (127.18.222.1:33889) I0918 20:38:11.785732 19405 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42534 I0918 20:38:11.790422 19584 heartbeater.cc:322] Connected to a master server at 127.18.222.60:39395 I0918 20:38:11.790848 19584 heartbeater.cc:406] Registering TS with master... I0918 20:38:11.791358 19584 heartbeater.cc:455] Master 127.18.222.60:39395 requested a full tablet report, sending... I0918 20:38:11.793129 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 123092us I0918 20:38:11.793473 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:11.793617 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:11.793814 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:11.793922 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:11.794653 19462 ts_manager.cc:186] Registered new tserver with Master: 37f57f2c32fe437fa6df8bf99c0a7610 (127.18.222.1:33889) I0918 20:38:11.794880 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/ts-1-root/instance: uuid: "244388b2e0714e6d81955c8eb0f32079" format_stamp: "Formatted at 2019-09-18 20:38:11 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:11.799890 19587 heartbeater.cc:322] Connected to a master server at 127.18.222.62:44223 I0918 20:38:11.800254 19587 heartbeater.cc:406] Registering TS with master... I0918 20:38:11.800760 19587 heartbeater.cc:455] Master 127.18.222.62:44223 requested a full tablet report, sending... I0918 20:38:11.801429 19585 heartbeater.cc:447] Master 127.18.222.61:43611 was elected leader, sending a full tablet report... I0918 20:38:11.803731 19348 ts_manager.cc:186] Registered new tserver with Master: 37f57f2c32fe437fa6df8bf99c0a7610 (127.18.222.1:33889) I0918 20:38:11.813079 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.018s user 0.014s sys 0.003s I0918 20:38:11.814839 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:11.825855 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:38:11.826061 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/ts-1-root uuid: "244388b2e0714e6d81955c8eb0f32079" format_stamp: "Formatted at 2019-09-18 20:38:11 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:11.826313 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:11.887029 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:11.887356 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:12.025231 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.2:40105 I0918 20:38:12.032263 19320 webserver.cc:374] Webserver started at http://127.18.222.2:45037/ using document root and password file I0918 20:38:12.128391 19655 heartbeater.cc:322] Connected to a master server at 127.18.222.60:39395 I0918 20:38:12.128862 19655 heartbeater.cc:406] Registering TS with master... I0918 20:38:12.149302 19655 heartbeater.cc:455] Master 127.18.222.60:39395 requested a full tablet report, sending... I0918 20:38:12.152539 19462 ts_manager.cc:186] Registered new tserver with Master: 244388b2e0714e6d81955c8eb0f32079 (127.18.222.2:40105) I0918 20:38:12.160522 19659 heartbeater.cc:322] Connected to a master server at 127.18.222.62:44223 I0918 20:38:12.160929 19659 heartbeater.cc:406] Registering TS with master... I0918 20:38:12.161499 19659 heartbeater.cc:455] Master 127.18.222.62:44223 requested a full tablet report, sending... I0918 20:38:12.164036 19348 ts_manager.cc:186] Registered new tserver with Master: 244388b2e0714e6d81955c8eb0f32079 (127.18.222.2:40105) I0918 20:38:12.174918 19656 heartbeater.cc:322] Connected to a master server at 127.18.222.61:43611 I0918 20:38:12.175276 19656 heartbeater.cc:406] Registering TS with master... I0918 20:38:12.175947 19656 heartbeater.cc:455] Master 127.18.222.61:43611 requested a full tablet report, sending... I0918 20:38:12.177812 19405 ts_manager.cc:186] Registered new tserver with Master: 244388b2e0714e6d81955c8eb0f32079 (127.18.222.2:40105) I0918 20:38:12.179914 19405 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42540 I0918 20:38:12.189110 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 123592us I0918 20:38:12.189617 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:12.189837 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:12.190115 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:12.190306 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:12.191640 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/ts-2-root/instance: uuid: "9568719bcc02432ebaea2dbd7568a4d2" format_stamp: "Formatted at 2019-09-18 20:38:12 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:12.192595 19656 heartbeater.cc:447] Master 127.18.222.61:43611 was elected leader, sending a full tablet report... I0918 20:38:12.208922 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.017s user 0.012s sys 0.004s I0918 20:38:12.211093 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:12.228945 19320 fs_manager.cc:419] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s I0918 20:38:12.229236 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/ts-2-root uuid: "9568719bcc02432ebaea2dbd7568a4d2" format_stamp: "Formatted at 2019-09-18 20:38:12 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:12.229548 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1568839089958813-19320/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:12.297559 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:12.297832 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:12.378075 19504 catalog_manager.cc:1153] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a: acquiring CA information for follower catalog manager: success I0918 20:38:12.382253 19504 catalog_manager.cc:1181] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:38:12.422255 19507 catalog_manager.cc:1153] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965: acquiring CA information for follower catalog manager: success I0918 20:38:12.426568 19507 catalog_manager.cc:1181] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:38:12.568435 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.3:46017 I0918 20:38:12.571882 19320 webserver.cc:374] Webserver started at http://127.18.222.3:38237/ using document root and password file I0918 20:38:12.669701 19730 heartbeater.cc:322] Connected to a master server at 127.18.222.62:44223 I0918 20:38:12.670089 19730 heartbeater.cc:406] Registering TS with master... I0918 20:38:12.670658 19730 heartbeater.cc:455] Master 127.18.222.62:44223 requested a full tablet report, sending... I0918 20:38:12.671522 19727 heartbeater.cc:322] Connected to a master server at 127.18.222.61:43611 I0918 20:38:12.671932 19727 heartbeater.cc:406] Registering TS with master... I0918 20:38:12.672425 19727 heartbeater.cc:455] Master 127.18.222.61:43611 requested a full tablet report, sending... I0918 20:38:12.673748 19348 ts_manager.cc:186] Registered new tserver with Master: 9568719bcc02432ebaea2dbd7568a4d2 (127.18.222.3:46017) I0918 20:38:12.676177 19404 ts_manager.cc:186] Registered new tserver with Master: 9568719bcc02432ebaea2dbd7568a4d2 (127.18.222.3:46017) I0918 20:38:12.678761 19726 heartbeater.cc:322] Connected to a master server at 127.18.222.60:39395 I0918 20:38:12.679111 19726 heartbeater.cc:406] Registering TS with master... I0918 20:38:12.679932 19726 heartbeater.cc:455] Master 127.18.222.60:39395 requested a full tablet report, sending... I0918 20:38:12.681463 19404 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42546 I0918 20:38:12.685567 19320 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.039991485s I0918 20:38:12.684710 19462 ts_manager.cc:186] Registered new tserver with Master: 9568719bcc02432ebaea2dbd7568a4d2 (127.18.222.3:46017) I0918 20:38:12.696283 19727 heartbeater.cc:447] Master 127.18.222.61:43611 was elected leader, sending a full tablet report... I0918 20:38:12.864734 19405 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42550: name: "ksck-test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\n\000\000\000\004\001\024\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:38:12.894269 19405 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:38:13.266258 19693 tablet_service.cc:1022] Processing CreateTablet for tablet 8b5822f647d84454aae1d2764a137c54 (table=ksck-test-table [id=f30bbb27eed640ef84d6ba50b8c1f9fc]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:13.267763 19693 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 8b5822f647d84454aae1d2764a137c54. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:13.271559 19692 tablet_service.cc:1022] Processing CreateTablet for tablet f9e633149efb42098ddfed9bf8dffad3 (table=ksck-test-table [id=f30bbb27eed640ef84d6ba50b8c1f9fc]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:13.273671 19691 tablet_service.cc:1022] Processing CreateTablet for tablet 97c5cc6796a841ea9c7ff4f235634ff2 (table=ksck-test-table [id=f30bbb27eed640ef84d6ba50b8c1f9fc]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:13.342867 19692 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f9e633149efb42098ddfed9bf8dffad3. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:13.356228 19622 tablet_service.cc:1022] Processing CreateTablet for tablet 8b5822f647d84454aae1d2764a137c54 (table=ksck-test-table [id=f30bbb27eed640ef84d6ba50b8c1f9fc]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:13.357309 19622 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 8b5822f647d84454aae1d2764a137c54. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:13.359654 19621 tablet_service.cc:1022] Processing CreateTablet for tablet f9e633149efb42098ddfed9bf8dffad3 (table=ksck-test-table [id=f30bbb27eed640ef84d6ba50b8c1f9fc]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:13.360790 19621 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f9e633149efb42098ddfed9bf8dffad3. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:13.364727 19620 tablet_service.cc:1022] Processing CreateTablet for tablet 97c5cc6796a841ea9c7ff4f235634ff2 (table=ksck-test-table [id=f30bbb27eed640ef84d6ba50b8c1f9fc]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:13.365952 19549 tablet_service.cc:1022] Processing CreateTablet for tablet 97c5cc6796a841ea9c7ff4f235634ff2 (table=ksck-test-table [id=f30bbb27eed640ef84d6ba50b8c1f9fc]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:13.367003 19549 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 97c5cc6796a841ea9c7ff4f235634ff2. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:13.371248 19620 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 97c5cc6796a841ea9c7ff4f235634ff2. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:13.378087 19691 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 97c5cc6796a841ea9c7ff4f235634ff2. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:13.395195 19550 tablet_service.cc:1022] Processing CreateTablet for tablet f9e633149efb42098ddfed9bf8dffad3 (table=ksck-test-table [id=f30bbb27eed640ef84d6ba50b8c1f9fc]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:13.398931 19551 tablet_service.cc:1022] Processing CreateTablet for tablet 8b5822f647d84454aae1d2764a137c54 (table=ksck-test-table [id=f30bbb27eed640ef84d6ba50b8c1f9fc]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:13.402318 19550 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f9e633149efb42098ddfed9bf8dffad3. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:13.407634 19551 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 8b5822f647d84454aae1d2764a137c54. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:13.440955 19758 tablet_bootstrap.cc:439] T f9e633149efb42098ddfed9bf8dffad3 P 9568719bcc02432ebaea2dbd7568a4d2: Bootstrap starting. I0918 20:38:13.507375 19760 tablet_bootstrap.cc:439] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610: Bootstrap starting. I0918 20:38:13.523280 19759 tablet_bootstrap.cc:439] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079: Bootstrap starting. I0918 20:38:13.523934 19758 tablet_bootstrap.cc:590] T f9e633149efb42098ddfed9bf8dffad3 P 9568719bcc02432ebaea2dbd7568a4d2: No blocks or log segments found. Creating new log. I0918 20:38:13.535094 19760 tablet_bootstrap.cc:590] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610: No blocks or log segments found. Creating new log. I0918 20:38:13.562212 19760 tablet_bootstrap.cc:439] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610: No bootstrap required, opened a new log I0918 20:38:13.562728 19760 ts_tablet_manager.cc:1085] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610: Time spent bootstrapping tablet: real 0.056s user 0.010s sys 0.015s I0918 20:38:13.563992 19759 tablet_bootstrap.cc:590] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079: No blocks or log segments found. Creating new log. I0918 20:38:13.577339 19758 tablet_bootstrap.cc:439] T f9e633149efb42098ddfed9bf8dffad3 P 9568719bcc02432ebaea2dbd7568a4d2: No bootstrap required, opened a new log I0918 20:38:13.578058 19758 ts_tablet_manager.cc:1085] T f9e633149efb42098ddfed9bf8dffad3 P 9568719bcc02432ebaea2dbd7568a4d2: Time spent bootstrapping tablet: real 0.188s user 0.012s sys 0.000s I0918 20:38:13.572628 19760 raft_consensus.cc:340] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:13.578683 19760 raft_consensus.cc:366] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:13.579198 19760 raft_consensus.cc:705] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37f57f2c32fe437fa6df8bf99c0a7610, State: Initialized, Role: FOLLOWER I0918 20:38:13.581467 19760 consensus_queue.cc:229] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:13.582309 19758 raft_consensus.cc:340] T f9e633149efb42098ddfed9bf8dffad3 P 9568719bcc02432ebaea2dbd7568a4d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:13.583133 19758 raft_consensus.cc:366] T f9e633149efb42098ddfed9bf8dffad3 P 9568719bcc02432ebaea2dbd7568a4d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:13.583472 19758 raft_consensus.cc:705] T f9e633149efb42098ddfed9bf8dffad3 P 9568719bcc02432ebaea2dbd7568a4d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9568719bcc02432ebaea2dbd7568a4d2, State: Initialized, Role: FOLLOWER I0918 20:38:13.584354 19758 consensus_queue.cc:229] T f9e633149efb42098ddfed9bf8dffad3 P 9568719bcc02432ebaea2dbd7568a4d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:13.611627 19760 ts_tablet_manager.cc:1113] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610: Time spent starting tablet: real 0.048s user 0.014s sys 0.005s I0918 20:38:13.710867 19760 tablet_bootstrap.cc:439] T f9e633149efb42098ddfed9bf8dffad3 P 37f57f2c32fe437fa6df8bf99c0a7610: Bootstrap starting. I0918 20:38:13.738857 19758 ts_tablet_manager.cc:1113] T f9e633149efb42098ddfed9bf8dffad3 P 9568719bcc02432ebaea2dbd7568a4d2: Time spent starting tablet: real 0.160s user 0.014s sys 0.023s I0918 20:38:13.739140 19759 tablet_bootstrap.cc:439] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079: No bootstrap required, opened a new log I0918 20:38:13.742719 19759 ts_tablet_manager.cc:1085] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079: Time spent bootstrapping tablet: real 0.228s user 0.009s sys 0.021s I0918 20:38:13.747936 19758 tablet_bootstrap.cc:439] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2: Bootstrap starting. I0918 20:38:13.750675 19759 raft_consensus.cc:340] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:13.751508 19759 raft_consensus.cc:366] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:13.755086 19759 raft_consensus.cc:705] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244388b2e0714e6d81955c8eb0f32079, State: Initialized, Role: FOLLOWER I0918 20:38:13.750250 19760 tablet_bootstrap.cc:590] T f9e633149efb42098ddfed9bf8dffad3 P 37f57f2c32fe437fa6df8bf99c0a7610: No blocks or log segments found. Creating new log. I0918 20:38:13.756422 19759 consensus_queue.cc:229] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:13.760434 19760 tablet_bootstrap.cc:439] T f9e633149efb42098ddfed9bf8dffad3 P 37f57f2c32fe437fa6df8bf99c0a7610: No bootstrap required, opened a new log I0918 20:38:13.765903 19760 ts_tablet_manager.cc:1085] T f9e633149efb42098ddfed9bf8dffad3 P 37f57f2c32fe437fa6df8bf99c0a7610: Time spent bootstrapping tablet: real 0.055s user 0.006s sys 0.002s I0918 20:38:13.768296 19760 raft_consensus.cc:340] T f9e633149efb42098ddfed9bf8dffad3 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:13.769171 19760 raft_consensus.cc:366] T f9e633149efb42098ddfed9bf8dffad3 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:13.769436 19760 raft_consensus.cc:705] T f9e633149efb42098ddfed9bf8dffad3 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37f57f2c32fe437fa6df8bf99c0a7610, State: Initialized, Role: FOLLOWER I0918 20:38:13.770175 19760 consensus_queue.cc:229] T f9e633149efb42098ddfed9bf8dffad3 P 37f57f2c32fe437fa6df8bf99c0a7610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:13.771194 19758 tablet_bootstrap.cc:590] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2: No blocks or log segments found. Creating new log. I0918 20:38:13.772001 19759 ts_tablet_manager.cc:1113] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079: Time spent starting tablet: real 0.029s user 0.006s sys 0.002s I0918 20:38:13.774165 19759 tablet_bootstrap.cc:439] T 8b5822f647d84454aae1d2764a137c54 P 244388b2e0714e6d81955c8eb0f32079: Bootstrap starting. I0918 20:38:13.853885 19760 ts_tablet_manager.cc:1113] T f9e633149efb42098ddfed9bf8dffad3 P 37f57f2c32fe437fa6df8bf99c0a7610: Time spent starting tablet: real 0.088s user 0.018s sys 0.005s I0918 20:38:13.855219 19760 tablet_bootstrap.cc:439] T 8b5822f647d84454aae1d2764a137c54 P 37f57f2c32fe437fa6df8bf99c0a7610: Bootstrap starting. I0918 20:38:13.858639 19767 raft_consensus.cc:472] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:13.859627 19767 raft_consensus.cc:494] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:13.862736 19767 leader_election.cc:296] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37f57f2c32fe437fa6df8bf99c0a7610 (127.18.222.1:33889), 9568719bcc02432ebaea2dbd7568a4d2 (127.18.222.3:46017) I0918 20:38:13.878990 19760 tablet_bootstrap.cc:590] T 8b5822f647d84454aae1d2764a137c54 P 37f57f2c32fe437fa6df8bf99c0a7610: No blocks or log segments found. Creating new log. I0918 20:38:13.885538 19759 tablet_bootstrap.cc:590] T 8b5822f647d84454aae1d2764a137c54 P 244388b2e0714e6d81955c8eb0f32079: No blocks or log segments found. Creating new log. I0918 20:38:13.890266 19758 tablet_bootstrap.cc:439] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2: No bootstrap required, opened a new log I0918 20:38:13.892248 19763 raft_consensus.cc:472] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:13.894273 19763 raft_consensus.cc:494] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:13.898571 19758 ts_tablet_manager.cc:1085] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2: Time spent bootstrapping tablet: real 0.152s user 0.011s sys 0.021s I0918 20:38:13.898561 19763 leader_election.cc:296] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244388b2e0714e6d81955c8eb0f32079 (127.18.222.2:40105), 9568719bcc02432ebaea2dbd7568a4d2 (127.18.222.3:46017) I0918 20:38:13.962931 19758 raft_consensus.cc:340] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:13.964807 19758 raft_consensus.cc:366] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:13.970705 19758 raft_consensus.cc:705] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9568719bcc02432ebaea2dbd7568a4d2, State: Initialized, Role: FOLLOWER I0918 20:38:14.054849 19759 tablet_bootstrap.cc:439] T 8b5822f647d84454aae1d2764a137c54 P 244388b2e0714e6d81955c8eb0f32079: No bootstrap required, opened a new log I0918 20:38:14.055344 19759 ts_tablet_manager.cc:1085] T 8b5822f647d84454aae1d2764a137c54 P 244388b2e0714e6d81955c8eb0f32079: Time spent bootstrapping tablet: real 0.283s user 0.010s sys 0.011s I0918 20:38:14.058688 19759 raft_consensus.cc:340] T 8b5822f647d84454aae1d2764a137c54 P 244388b2e0714e6d81955c8eb0f32079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:14.059355 19759 raft_consensus.cc:366] T 8b5822f647d84454aae1d2764a137c54 P 244388b2e0714e6d81955c8eb0f32079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:14.059545 19759 raft_consensus.cc:705] T 8b5822f647d84454aae1d2764a137c54 P 244388b2e0714e6d81955c8eb0f32079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244388b2e0714e6d81955c8eb0f32079, State: Initialized, Role: FOLLOWER I0918 20:38:14.060322 19759 consensus_queue.cc:229] T 8b5822f647d84454aae1d2764a137c54 P 244388b2e0714e6d81955c8eb0f32079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:14.063460 19759 ts_tablet_manager.cc:1113] T 8b5822f647d84454aae1d2764a137c54 P 244388b2e0714e6d81955c8eb0f32079: Time spent starting tablet: real 0.008s user 0.001s sys 0.005s I0918 20:38:14.053825 19758 consensus_queue.cc:229] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:14.064342 19759 tablet_bootstrap.cc:439] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079: Bootstrap starting. I0918 20:38:14.065976 19758 ts_tablet_manager.cc:1113] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2: Time spent starting tablet: real 0.160s user 0.009s sys 0.002s I0918 20:38:14.066778 19758 tablet_bootstrap.cc:439] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2: Bootstrap starting. I0918 20:38:14.081339 19765 raft_consensus.cc:472] T f9e633149efb42098ddfed9bf8dffad3 P 9568719bcc02432ebaea2dbd7568a4d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:14.081995 19765 raft_consensus.cc:494] T f9e633149efb42098ddfed9bf8dffad3 P 9568719bcc02432ebaea2dbd7568a4d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:14.084002 19561 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f9e633149efb42098ddfed9bf8dffad3" candidate_uuid: "244388b2e0714e6d81955c8eb0f32079" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" is_pre_election: true I0918 20:38:14.085258 19765 leader_election.cc:296] T f9e633149efb42098ddfed9bf8dffad3 P 9568719bcc02432ebaea2dbd7568a4d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37f57f2c32fe437fa6df8bf99c0a7610 (127.18.222.1:33889), 244388b2e0714e6d81955c8eb0f32079 (127.18.222.2:40105) I0918 20:38:14.085337 19561 raft_consensus.cc:2350] T f9e633149efb42098ddfed9bf8dffad3 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244388b2e0714e6d81955c8eb0f32079 in term 0. I0918 20:38:14.085423 19760 tablet_bootstrap.cc:439] T 8b5822f647d84454aae1d2764a137c54 P 37f57f2c32fe437fa6df8bf99c0a7610: No bootstrap required, opened a new log I0918 20:38:14.108695 19765 raft_consensus.cc:472] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:14.110508 19765 raft_consensus.cc:494] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:14.112843 19765 leader_election.cc:296] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37f57f2c32fe437fa6df8bf99c0a7610 (127.18.222.1:33889), 244388b2e0714e6d81955c8eb0f32079 (127.18.222.2:40105) I0918 20:38:14.108201 19760 ts_tablet_manager.cc:1085] T 8b5822f647d84454aae1d2764a137c54 P 37f57f2c32fe437fa6df8bf99c0a7610: Time spent bootstrapping tablet: real 0.253s user 0.012s sys 0.005s I0918 20:38:14.130061 19759 tablet_bootstrap.cc:590] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079: No blocks or log segments found. Creating new log. I0918 20:38:14.131374 19596 leader_election.cc:310] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244388b2e0714e6d81955c8eb0f32079, 37f57f2c32fe437fa6df8bf99c0a7610; no voters: I0918 20:38:14.133785 19767 raft_consensus.cc:2644] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:14.134173 19767 raft_consensus.cc:472] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:14.134428 19767 raft_consensus.cc:2890] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:14.134019 19760 raft_consensus.cc:340] T 8b5822f647d84454aae1d2764a137c54 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:14.138588 19760 raft_consensus.cc:366] T 8b5822f647d84454aae1d2764a137c54 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:14.138851 19760 raft_consensus.cc:705] T 8b5822f647d84454aae1d2764a137c54 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37f57f2c32fe437fa6df8bf99c0a7610, State: Initialized, Role: FOLLOWER I0918 20:38:14.139566 19760 consensus_queue.cc:229] T 8b5822f647d84454aae1d2764a137c54 P 37f57f2c32fe437fa6df8bf99c0a7610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:14.154359 19760 ts_tablet_manager.cc:1113] T 8b5822f647d84454aae1d2764a137c54 P 37f57f2c32fe437fa6df8bf99c0a7610: Time spent starting tablet: real 0.040s user 0.006s sys 0.000s I0918 20:38:14.160868 19758 tablet_bootstrap.cc:590] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2: No blocks or log segments found. Creating new log. I0918 20:38:14.169085 19703 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f9e633149efb42098ddfed9bf8dffad3" candidate_uuid: "244388b2e0714e6d81955c8eb0f32079" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9568719bcc02432ebaea2dbd7568a4d2" is_pre_election: true I0918 20:38:14.173169 19703 raft_consensus.cc:2350] T f9e633149efb42098ddfed9bf8dffad3 P 9568719bcc02432ebaea2dbd7568a4d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244388b2e0714e6d81955c8eb0f32079 in term 0. I0918 20:38:14.198025 19767 raft_consensus.cc:494] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:14.199983 19767 leader_election.cc:296] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [CANDIDATE]: Term 1 election: Requested vote from peers 37f57f2c32fe437fa6df8bf99c0a7610 (127.18.222.1:33889), 9568719bcc02432ebaea2dbd7568a4d2 (127.18.222.3:46017) I0918 20:38:14.205301 19703 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f9e633149efb42098ddfed9bf8dffad3" candidate_uuid: "244388b2e0714e6d81955c8eb0f32079" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9568719bcc02432ebaea2dbd7568a4d2" I0918 20:38:14.205991 19703 raft_consensus.cc:2890] T f9e633149efb42098ddfed9bf8dffad3 P 9568719bcc02432ebaea2dbd7568a4d2 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:14.212586 19561 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f9e633149efb42098ddfed9bf8dffad3" candidate_uuid: "244388b2e0714e6d81955c8eb0f32079" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" I0918 20:38:14.213217 19561 raft_consensus.cc:2890] T f9e633149efb42098ddfed9bf8dffad3 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:14.226116 19561 raft_consensus.cc:2350] T f9e633149efb42098ddfed9bf8dffad3 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244388b2e0714e6d81955c8eb0f32079 in term 1. I0918 20:38:14.227556 19596 leader_election.cc:310] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244388b2e0714e6d81955c8eb0f32079, 37f57f2c32fe437fa6df8bf99c0a7610; no voters: I0918 20:38:14.229212 19767 raft_consensus.cc:2644] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:14.257913 19767 raft_consensus.cc:667] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [term 1 LEADER]: Becoming Leader. State: Replica: 244388b2e0714e6d81955c8eb0f32079, State: Running, Role: LEADER I0918 20:38:14.269032 19632 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "97c5cc6796a841ea9c7ff4f235634ff2" candidate_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244388b2e0714e6d81955c8eb0f32079" is_pre_election: true I0918 20:38:14.264956 19767 consensus_queue.cc:206] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } W0918 20:38:14.274720 19510 leader_election.cc:348] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244388b2e0714e6d81955c8eb0f32079 (127.18.222.2:40105): Illegal state: must be running to vote when last-logged opid is not known I0918 20:38:14.274989 19759 tablet_bootstrap.cc:439] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079: No bootstrap required, opened a new log I0918 20:38:14.278730 19759 ts_tablet_manager.cc:1085] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079: Time spent bootstrapping tablet: real 0.215s user 0.027s sys 0.028s I0918 20:38:14.278371 19758 tablet_bootstrap.cc:439] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2: No bootstrap required, opened a new log W0918 20:38:14.287236 19517 debug-util.cc:397] Leaking SignalData structure 0x7b0800117060 after lost signal to thread 19323 I0918 20:38:14.286852 19759 raft_consensus.cc:340] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:14.287817 19703 raft_consensus.cc:2350] T f9e633149efb42098ddfed9bf8dffad3 P 9568719bcc02432ebaea2dbd7568a4d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244388b2e0714e6d81955c8eb0f32079 in term 1. I0918 20:38:14.288178 19759 raft_consensus.cc:366] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:14.288516 19759 raft_consensus.cc:705] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244388b2e0714e6d81955c8eb0f32079, State: Initialized, Role: FOLLOWER I0918 20:38:14.289220 19758 ts_tablet_manager.cc:1085] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2: Time spent bootstrapping tablet: real 0.223s user 0.034s sys 0.032s W0918 20:38:14.288492 19517 debug-util.cc:397] Leaking SignalData structure 0x7b0800117720 after lost signal to thread 19379 I0918 20:38:14.295413 19758 raft_consensus.cc:340] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:14.299579 19758 raft_consensus.cc:366] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:14.300030 19758 raft_consensus.cc:705] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9568719bcc02432ebaea2dbd7568a4d2, State: Initialized, Role: FOLLOWER I0918 20:38:14.292209 19759 consensus_queue.cc:229] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:14.303071 19759 ts_tablet_manager.cc:1113] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079: Time spent starting tablet: real 0.023s user 0.006s sys 0.002s W0918 20:38:14.292670 19517 debug-util.cc:397] Leaking SignalData structure 0x7b0800117de0 after lost signal to thread 19436 W0918 20:38:14.308146 19517 debug-util.cc:397] Leaking SignalData structure 0x7b08000fb0c0 after lost signal to thread 19493 I0918 20:38:14.323132 19632 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f9e633149efb42098ddfed9bf8dffad3" candidate_uuid: "9568719bcc02432ebaea2dbd7568a4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244388b2e0714e6d81955c8eb0f32079" is_pre_election: true I0918 20:38:14.324589 19631 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8b5822f647d84454aae1d2764a137c54" candidate_uuid: "9568719bcc02432ebaea2dbd7568a4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244388b2e0714e6d81955c8eb0f32079" is_pre_election: true I0918 20:38:14.326155 19631 raft_consensus.cc:2350] T 8b5822f647d84454aae1d2764a137c54 P 244388b2e0714e6d81955c8eb0f32079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9568719bcc02432ebaea2dbd7568a4d2 in term 0. I0918 20:38:14.306756 19703 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "97c5cc6796a841ea9c7ff4f235634ff2" candidate_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9568719bcc02432ebaea2dbd7568a4d2" is_pre_election: true I0918 20:38:14.328467 19632 raft_consensus.cc:2311] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 9568719bcc02432ebaea2dbd7568a4d2 for term 1 because replica is either leader or believes a valid leader to be alive. I0918 20:38:14.328811 19664 leader_election.cc:310] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244388b2e0714e6d81955c8eb0f32079, 9568719bcc02432ebaea2dbd7568a4d2; no voters: I0918 20:38:14.330102 19765 raft_consensus.cc:2644] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:14.330401 19765 raft_consensus.cc:472] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:14.330693 19765 raft_consensus.cc:2890] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:14.306638 19758 consensus_queue.cc:229] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:14.334684 19703 raft_consensus.cc:2350] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37f57f2c32fe437fa6df8bf99c0a7610 in term 0. I0918 20:38:14.336380 19511 leader_election.cc:310] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37f57f2c32fe437fa6df8bf99c0a7610, 9568719bcc02432ebaea2dbd7568a4d2; no voters: 244388b2e0714e6d81955c8eb0f32079 I0918 20:38:14.338937 19763 raft_consensus.cc:2644] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:14.339226 19763 raft_consensus.cc:472] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:14.339529 19763 raft_consensus.cc:2890] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:14.358801 19765 raft_consensus.cc:494] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:14.361101 19763 raft_consensus.cc:494] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:14.363543 19632 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8b5822f647d84454aae1d2764a137c54" candidate_uuid: "9568719bcc02432ebaea2dbd7568a4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244388b2e0714e6d81955c8eb0f32079" I0918 20:38:14.365027 19632 raft_consensus.cc:2890] T 8b5822f647d84454aae1d2764a137c54 P 244388b2e0714e6d81955c8eb0f32079 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:14.365345 19763 leader_election.cc:296] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [CANDIDATE]: Term 1 election: Requested vote from peers 244388b2e0714e6d81955c8eb0f32079 (127.18.222.2:40105), 9568719bcc02432ebaea2dbd7568a4d2 (127.18.222.3:46017) I0918 20:38:14.368382 19631 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "97c5cc6796a841ea9c7ff4f235634ff2" candidate_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244388b2e0714e6d81955c8eb0f32079" I0918 20:38:14.370841 19631 raft_consensus.cc:2890] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:14.366592 19765 leader_election.cc:296] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 [CANDIDATE]: Term 1 election: Requested vote from peers 37f57f2c32fe437fa6df8bf99c0a7610 (127.18.222.1:33889), 244388b2e0714e6d81955c8eb0f32079 (127.18.222.2:40105) I0918 20:38:14.367590 19703 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "97c5cc6796a841ea9c7ff4f235634ff2" candidate_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9568719bcc02432ebaea2dbd7568a4d2" I0918 20:38:14.386620 19632 raft_consensus.cc:2350] T 8b5822f647d84454aae1d2764a137c54 P 244388b2e0714e6d81955c8eb0f32079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9568719bcc02432ebaea2dbd7568a4d2 in term 1. I0918 20:38:14.387902 19664 leader_election.cc:310] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244388b2e0714e6d81955c8eb0f32079, 9568719bcc02432ebaea2dbd7568a4d2; no voters: I0918 20:38:14.388458 19703 raft_consensus.cc:2890] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2 [term 0 FOLLOWER]: Advancing to term 1 W0918 20:38:14.393280 19785 log.cc:669] Time spent T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079: Append to log took a long time: real 0.055s user 0.001s sys 0.000s I0918 20:38:14.398018 19758 ts_tablet_manager.cc:1113] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2: Time spent starting tablet: real 0.108s user 0.021s sys 0.014s I0918 20:38:14.397447 19765 raft_consensus.cc:2644] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:14.399518 19765 raft_consensus.cc:667] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 [term 1 LEADER]: Becoming Leader. State: Replica: 9568719bcc02432ebaea2dbd7568a4d2, State: Running, Role: LEADER I0918 20:38:14.389128 19631 raft_consensus.cc:2350] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37f57f2c32fe437fa6df8bf99c0a7610 in term 1. I0918 20:38:14.401019 19765 consensus_queue.cc:206] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:14.404121 19510 leader_election.cc:310] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244388b2e0714e6d81955c8eb0f32079, 37f57f2c32fe437fa6df8bf99c0a7610; no voters: I0918 20:38:14.405979 19763 raft_consensus.cc:2644] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:14.414330 19763 raft_consensus.cc:667] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 1 LEADER]: Becoming Leader. State: Replica: 37f57f2c32fe437fa6df8bf99c0a7610, State: Running, Role: LEADER I0918 20:38:14.403919 19402 catalog_manager.cc:4131] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 reported cstate change: term changed from 0 to 1, leader changed from to 244388b2e0714e6d81955c8eb0f32079 (127.18.222.2). New cstate: current_term: 1 leader_uuid: "244388b2e0714e6d81955c8eb0f32079" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } health_report { overall_health: UNKNOWN } } } I0918 20:38:14.421770 19703 raft_consensus.cc:2350] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37f57f2c32fe437fa6df8bf99c0a7610 in term 1. I0918 20:38:14.416404 19763 consensus_queue.cc:206] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:14.435068 19403 catalog_manager.cc:4131] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 reported cstate change: term changed from 0 to 1, leader changed from to 9568719bcc02432ebaea2dbd7568a4d2 (127.18.222.3). New cstate: current_term: 1 leader_uuid: "9568719bcc02432ebaea2dbd7568a4d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } health_report { overall_health: HEALTHY } } } I0918 20:38:14.442915 19559 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8b5822f647d84454aae1d2764a137c54" candidate_uuid: "9568719bcc02432ebaea2dbd7568a4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" I0918 20:38:14.441285 19561 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f9e633149efb42098ddfed9bf8dffad3" candidate_uuid: "9568719bcc02432ebaea2dbd7568a4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" is_pre_election: true I0918 20:38:14.444667 19559 raft_consensus.cc:2890] T 8b5822f647d84454aae1d2764a137c54 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:14.443876 19560 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8b5822f647d84454aae1d2764a137c54" candidate_uuid: "9568719bcc02432ebaea2dbd7568a4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" is_pre_election: true I0918 20:38:14.446035 19561 raft_consensus.cc:2280] T f9e633149efb42098ddfed9bf8dffad3 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9568719bcc02432ebaea2dbd7568a4d2 in current term 1: Already voted for candidate 244388b2e0714e6d81955c8eb0f32079 in this term. I0918 20:38:14.452175 19667 leader_election.cc:310] T f9e633149efb42098ddfed9bf8dffad3 P 9568719bcc02432ebaea2dbd7568a4d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9568719bcc02432ebaea2dbd7568a4d2; no voters: 244388b2e0714e6d81955c8eb0f32079, 37f57f2c32fe437fa6df8bf99c0a7610 I0918 20:38:14.453948 19765 raft_consensus.cc:2592] T f9e633149efb42098ddfed9bf8dffad3 P 9568719bcc02432ebaea2dbd7568a4d2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0918 20:38:14.461249 19559 raft_consensus.cc:2350] T 8b5822f647d84454aae1d2764a137c54 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9568719bcc02432ebaea2dbd7568a4d2 in term 1. I0918 20:38:14.462024 19560 raft_consensus.cc:2325] T 8b5822f647d84454aae1d2764a137c54 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9568719bcc02432ebaea2dbd7568a4d2 for term 1 because replica is already servicing an update from a current leader or another vote. I0918 20:38:14.494395 19404 catalog_manager.cc:4131] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 reported cstate change: term changed from 0 to 1, leader changed from to 37f57f2c32fe437fa6df8bf99c0a7610 (127.18.222.1). New cstate: current_term: 1 leader_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } health_report { overall_health: UNKNOWN } } } I0918 20:38:14.811347 19767 consensus_queue.cc:984] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:14.854634 19765 consensus_queue.cc:984] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:14.894708 19763 consensus_queue.cc:984] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:14.895378 19787 consensus_queue.cc:984] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:14.897382 19767 consensus_queue.cc:984] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:14.955796 19765 consensus_queue.cc:984] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s W0918 20:38:15.156566 19816 log.cc:669] Time spent T f9e633149efb42098ddfed9bf8dffad3 P 37f57f2c32fe437fa6df8bf99c0a7610: Append to log took a long time: real 0.097s user 0.001s sys 0.000s W0918 20:38:15.237356 19596 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.222.1:33889, user_credentials={real_user=slave}} blocked reactor thread for 56734.3us W0918 20:38:15.233839 19594 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.222.3:46017, user_credentials={real_user=slave}} blocked reactor thread for 53260.3us I0918 20:38:16.302183 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:16.393577 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:16.394089 19320 tablet_replica.cc:273] T 8b5822f647d84454aae1d2764a137c54 P 37f57f2c32fe437fa6df8bf99c0a7610: stopping tablet replica I0918 20:38:16.394773 19320 raft_consensus.cc:2147] T 8b5822f647d84454aae1d2764a137c54 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:16.395256 19320 raft_consensus.cc:2174] T 8b5822f647d84454aae1d2764a137c54 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:16.404067 19320 tablet_replica.cc:273] T f9e633149efb42098ddfed9bf8dffad3 P 37f57f2c32fe437fa6df8bf99c0a7610: stopping tablet replica I0918 20:38:16.404422 19320 raft_consensus.cc:2147] T f9e633149efb42098ddfed9bf8dffad3 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:16.404712 19320 raft_consensus.cc:2174] T f9e633149efb42098ddfed9bf8dffad3 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:16.406872 19320 tablet_replica.cc:273] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610: stopping tablet replica I0918 20:38:16.409816 19320 raft_consensus.cc:2147] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:16.412461 19320 raft_consensus.cc:2174] T 97c5cc6796a841ea9c7ff4f235634ff2 P 37f57f2c32fe437fa6df8bf99c0a7610 [term 1 FOLLOWER]: Raft consensus is shut down! W0918 20:38:16.716640 19667 consensus_peers.cc:458] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 -> Peer 37f57f2c32fe437fa6df8bf99c0a7610 (127.18.222.1:33889): Couldn't send request to peer 37f57f2c32fe437fa6df8bf99c0a7610. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.1:33889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0918 20:38:16.721133 19596 consensus_peers.cc:458] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 -> Peer 37f57f2c32fe437fa6df8bf99c0a7610 (127.18.222.1:33889): Couldn't send request to peer 37f57f2c32fe437fa6df8bf99c0a7610. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.1:33889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:38:17.638736 19862 raft_consensus.cc:472] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37f57f2c32fe437fa6df8bf99c0a7610) I0918 20:38:17.639173 19862 raft_consensus.cc:494] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:17.640547 19862 leader_election.cc:296] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 244388b2e0714e6d81955c8eb0f32079 (127.18.222.2:40105), 37f57f2c32fe437fa6df8bf99c0a7610 (127.18.222.1:33889) I0918 20:38:17.641393 19632 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "97c5cc6796a841ea9c7ff4f235634ff2" candidate_uuid: "9568719bcc02432ebaea2dbd7568a4d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "244388b2e0714e6d81955c8eb0f32079" is_pre_election: true I0918 20:38:17.642226 19632 raft_consensus.cc:2311] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9568719bcc02432ebaea2dbd7568a4d2 for term 2 because replica is either leader or believes a valid leader to be alive. W0918 20:38:17.642799 19667 leader_election.cc:341] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 37f57f2c32fe437fa6df8bf99c0a7610 (127.18.222.1:33889): Network error: Client connection negotiation failed: client connection to 127.18.222.1:33889: connect: Connection refused (error 111) I0918 20:38:17.643625 19664 leader_election.cc:310] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9568719bcc02432ebaea2dbd7568a4d2; no voters: 244388b2e0714e6d81955c8eb0f32079, 37f57f2c32fe437fa6df8bf99c0a7610 I0918 20:38:17.644830 19862 raft_consensus.cc:2592] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0918 20:38:17.657311 19861 raft_consensus.cc:472] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37f57f2c32fe437fa6df8bf99c0a7610) I0918 20:38:17.657855 19861 raft_consensus.cc:494] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:17.659642 19861 leader_election.cc:296] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37f57f2c32fe437fa6df8bf99c0a7610 (127.18.222.1:33889), 9568719bcc02432ebaea2dbd7568a4d2 (127.18.222.3:46017) I0918 20:38:17.660806 19702 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "97c5cc6796a841ea9c7ff4f235634ff2" candidate_uuid: "244388b2e0714e6d81955c8eb0f32079" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9568719bcc02432ebaea2dbd7568a4d2" is_pre_election: true I0918 20:38:17.661516 19702 raft_consensus.cc:2350] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244388b2e0714e6d81955c8eb0f32079 in term 1. W0918 20:38:17.662521 19596 leader_election.cc:341] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 37f57f2c32fe437fa6df8bf99c0a7610 (127.18.222.1:33889): Network error: Client connection negotiation failed: client connection to 127.18.222.1:33889: connect: Connection refused (error 111) I0918 20:38:17.664153 19594 leader_election.cc:310] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 244388b2e0714e6d81955c8eb0f32079, 9568719bcc02432ebaea2dbd7568a4d2; no voters: 37f57f2c32fe437fa6df8bf99c0a7610 I0918 20:38:17.665012 19861 raft_consensus.cc:2644] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0918 20:38:17.665343 19861 raft_consensus.cc:472] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 37f57f2c32fe437fa6df8bf99c0a7610) I0918 20:38:17.665714 19861 raft_consensus.cc:2890] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [term 1 FOLLOWER]: Advancing to term 2 I0918 20:38:17.673099 19861 raft_consensus.cc:494] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } W0918 20:38:17.677531 19596 leader_election.cc:341] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 37f57f2c32fe437fa6df8bf99c0a7610 (127.18.222.1:33889): Network error: Client connection negotiation failed: client connection to 127.18.222.1:33889: connect: Connection refused (error 111) I0918 20:38:17.678071 19861 leader_election.cc:296] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [CANDIDATE]: Term 2 election: Requested vote from peers 37f57f2c32fe437fa6df8bf99c0a7610 (127.18.222.1:33889), 9568719bcc02432ebaea2dbd7568a4d2 (127.18.222.3:46017) I0918 20:38:17.678944 19702 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "97c5cc6796a841ea9c7ff4f235634ff2" candidate_uuid: "244388b2e0714e6d81955c8eb0f32079" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9568719bcc02432ebaea2dbd7568a4d2" I0918 20:38:17.679539 19702 raft_consensus.cc:2890] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2 [term 1 FOLLOWER]: Advancing to term 2 I0918 20:38:17.686556 19702 raft_consensus.cc:2350] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244388b2e0714e6d81955c8eb0f32079 in term 2. I0918 20:38:17.687649 19594 leader_election.cc:310] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 244388b2e0714e6d81955c8eb0f32079, 9568719bcc02432ebaea2dbd7568a4d2; no voters: 37f57f2c32fe437fa6df8bf99c0a7610 I0918 20:38:17.688484 19861 raft_consensus.cc:2644] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [term 2 FOLLOWER]: Leader election won for term 2 I0918 20:38:17.692498 19861 raft_consensus.cc:667] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [term 2 LEADER]: Becoming Leader. State: Replica: 244388b2e0714e6d81955c8eb0f32079, State: Running, Role: LEADER I0918 20:38:17.693274 19861 consensus_queue.cc:206] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } } peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } } I0918 20:38:17.705708 19403 catalog_manager.cc:4131] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 reported cstate change: term changed from 1 to 2, leader changed from 37f57f2c32fe437fa6df8bf99c0a7610 (127.18.222.1) to 244388b2e0714e6d81955c8eb0f32079 (127.18.222.2). New cstate: current_term: 2 leader_uuid: "244388b2e0714e6d81955c8eb0f32079" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244388b2e0714e6d81955c8eb0f32079" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37f57f2c32fe437fa6df8bf99c0a7610" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 } health_report { overall_health: UNKNOWN } } } W0918 20:38:17.717882 19320 thread.cc:499] Waited for 1000ms trying to join with diag-logger (tid 19517) W0918 20:38:18.190979 19596 consensus_peers.cc:458] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 -> Peer 37f57f2c32fe437fa6df8bf99c0a7610 (127.18.222.1:33889): Couldn't send request to peer 37f57f2c32fe437fa6df8bf99c0a7610. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.1:33889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:38:18.290751 19702 raft_consensus.cc:1184] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2 [term 2 FOLLOWER]: Refusing update from remote peer 244388b2e0714e6d81955c8eb0f32079: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch) I0918 20:38:18.292882 19861 consensus_queue.cc:984] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9568719bcc02432ebaea2dbd7568a4d2" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s W0918 20:38:18.718359 19320 thread.cc:499] Waited for 2000ms trying to join with diag-logger (tid 19517) W0918 20:38:19.144130 19596 consensus_peers.cc:458] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 -> Peer 37f57f2c32fe437fa6df8bf99c0a7610 (127.18.222.1:33889): Couldn't send request to peer 37f57f2c32fe437fa6df8bf99c0a7610. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.1:33889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0918 20:38:19.371199 19667 consensus_peers.cc:458] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 -> Peer 37f57f2c32fe437fa6df8bf99c0a7610 (127.18.222.1:33889): Couldn't send request to peer 37f57f2c32fe437fa6df8bf99c0a7610. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.1:33889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0918 20:38:19.718798 19320 thread.cc:499] Waited for 3000ms trying to join with diag-logger (tid 19517) I0918 20:38:20.156601 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:20.156916 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:20.157398 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:20.176081 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:20.195637 19596 messenger.cc:381] Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer W0918 20:38:20.198114 19664 consensus_peers.cc:458] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 -> Peer 244388b2e0714e6d81955c8eb0f32079 (127.18.222.2:40105): Couldn't send request to peer 244388b2e0714e6d81955c8eb0f32079. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry. I0918 20:38:20.228312 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:20.228716 19320 tablet_replica.cc:273] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079: stopping tablet replica I0918 20:38:20.229846 19320 raft_consensus.cc:2147] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [term 2 LEADER]: Raft consensus shutting down. I0918 20:38:20.231161 19320 raft_consensus.cc:2174] T 97c5cc6796a841ea9c7ff4f235634ff2 P 244388b2e0714e6d81955c8eb0f32079 [term 2 FOLLOWER]: Raft consensus is shut down! I0918 20:38:20.233613 19320 tablet_replica.cc:273] T 8b5822f647d84454aae1d2764a137c54 P 244388b2e0714e6d81955c8eb0f32079: stopping tablet replica I0918 20:38:20.233952 19320 raft_consensus.cc:2147] T 8b5822f647d84454aae1d2764a137c54 P 244388b2e0714e6d81955c8eb0f32079 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:20.234268 19320 raft_consensus.cc:2174] T 8b5822f647d84454aae1d2764a137c54 P 244388b2e0714e6d81955c8eb0f32079 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:20.236248 19320 tablet_replica.cc:273] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079: stopping tablet replica I0918 20:38:20.236584 19320 raft_consensus.cc:2147] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:20.237802 19320 raft_consensus.cc:2174] T f9e633149efb42098ddfed9bf8dffad3 P 244388b2e0714e6d81955c8eb0f32079 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:20.262059 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:20.262249 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:20.262533 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:20.274516 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:20.307598 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:20.307986 19320 tablet_replica.cc:273] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2: stopping tablet replica I0918 20:38:20.308393 19320 raft_consensus.cc:2147] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2 [term 2 FOLLOWER]: Raft consensus shutting down. I0918 20:38:20.308686 19320 raft_consensus.cc:2174] T 97c5cc6796a841ea9c7ff4f235634ff2 P 9568719bcc02432ebaea2dbd7568a4d2 [term 2 FOLLOWER]: Raft consensus is shut down! I0918 20:38:20.310967 19320 tablet_replica.cc:273] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2: stopping tablet replica I0918 20:38:20.311318 19320 raft_consensus.cc:2147] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:20.312269 19320 raft_consensus.cc:2174] T 8b5822f647d84454aae1d2764a137c54 P 9568719bcc02432ebaea2dbd7568a4d2 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:20.314338 19320 tablet_replica.cc:273] T f9e633149efb42098ddfed9bf8dffad3 P 9568719bcc02432ebaea2dbd7568a4d2: stopping tablet replica I0918 20:38:20.314656 19320 raft_consensus.cc:2147] T f9e633149efb42098ddfed9bf8dffad3 P 9568719bcc02432ebaea2dbd7568a4d2 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:20.314932 19320 raft_consensus.cc:2174] T f9e633149efb42098ddfed9bf8dffad3 P 9568719bcc02432ebaea2dbd7568a4d2 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:20.336298 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:20.336479 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:20.336788 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:20.346868 19320 master.cc:247] Master@127.18.222.62:44223 shutting down... I0918 20:38:20.371186 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:20.371572 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:20.371868 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P a9903afeb95a4ed8b8110650f809f965: stopping tablet replica I0918 20:38:20.390218 19320 master.cc:258] Master@127.18.222.62:44223 shutdown complete. I0918 20:38:20.400136 19320 master.cc:247] Master@127.18.222.61:43611 shutting down... W0918 20:38:20.414322 19387 consensus_peers.cc:458] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 -> Peer a9903afeb95a4ed8b8110650f809f965 (127.18.222.62:44223): Couldn't send request to peer a9903afeb95a4ed8b8110650f809f965. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.62:44223: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:38:20.442407 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:20.443186 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:20.443460 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 200998a474bb42b2a13b5ec292154c78: stopping tablet replica I0918 20:38:20.462410 19320 master.cc:258] Master@127.18.222.61:43611 shutdown complete. I0918 20:38:20.473563 19320 master.cc:247] Master@127.18.222.60:39395 shutting down... I0918 20:38:20.496407 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:20.496767 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:20.496992 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P db6ca21578fe4f5a86e7d466226a447a: stopping tablet replica I0918 20:38:20.514798 19320 master.cc:258] Master@127.18.222.60:39395 shutdown complete. [ OK ] RemoteKsckTest.TestClusterOk (10557 ms) [ RUN ] RemoteKsckTest.TestCheckUnusualFlags I0918 20:38:20.553264 19320 test_util.cc:212] Using random seed: -783525866 I0918 20:38:20.554045 19320 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.18.222.62:33587,127.18.222.61:45425,127.18.222.60:44401 I0918 20:38:20.566510 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 127592us I0918 20:38:20.566890 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:20.567034 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:20.567282 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:20.567400 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:20.568533 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/master-0-root/instance: uuid: "5fa8ea8cc088448b988deacd9000a0ad" format_stamp: "Formatted at 2019-09-18 20:38:20 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:20.576819 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.008s sys 0.001s I0918 20:38:20.578217 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:20.585548 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:20.585729 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/master-0-root uuid: "5fa8ea8cc088448b988deacd9000a0ad" format_stamp: "Formatted at 2019-09-18 20:38:20 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:20.586000 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:20.662338 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:33587 I0918 20:38:20.664161 19320 webserver.cc:374] Webserver started at http://127.18.222.62:33487/ using document root and password file I0918 20:38:20.666028 19941 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:20.670600 19941 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:20.683794 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 127592us I0918 20:38:20.684293 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:20.684520 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:20.684839 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:20.685057 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:20.686448 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/master-1-root/instance: uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" format_stamp: "Formatted at 2019-09-18 20:38:20 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:20.688985 19941 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } has no permanent_uuid. Determining permanent_uuid... W0918 20:38:20.690621 19941 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:45425: Network error: Client connection negotiation failed: client connection to 127.18.222.61:45425: connect: Connection refused (error 111) I0918 20:38:20.696789 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.008s sys 0.003s I0918 20:38:20.698274 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:20.706135 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0918 20:38:20.706324 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/master-1-root uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" format_stamp: "Formatted at 2019-09-18 20:38:20 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:20.706550 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:20.749123 19941 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } attempt: 1 W0918 20:38:20.751389 19941 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:45425: Network error: Client connection negotiation failed: client connection to 127.18.222.61:45425: connect: Connection refused (error 111) I0918 20:38:20.758539 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:45425 I0918 20:38:20.760270 19320 webserver.cc:374] Webserver started at http://127.18.222.61:43179/ using document root and password file I0918 20:38:20.762051 19998 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:20.766223 19998 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:20.782130 19998 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:20.786242 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 127592us I0918 20:38:20.786799 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:20.787042 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:20.787360 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:20.787560 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:20.790531 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/master-2-root/instance: uuid: "252ccf1ea83e476392b25d8272d26a03" format_stamp: "Formatted at 2019-09-18 20:38:20 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:20.798749 19998 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } has no permanent_uuid. Determining permanent_uuid... W0918 20:38:20.800575 19998 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:44401: Network error: Client connection negotiation failed: client connection to 127.18.222.60:44401: connect: Connection refused (error 111) I0918 20:38:20.803282 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.012s user 0.013s sys 0.000s I0918 20:38:20.804888 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:20.813097 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s I0918 20:38:20.813313 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/master-2-root uuid: "252ccf1ea83e476392b25d8272d26a03" format_stamp: "Formatted at 2019-09-18 20:38:20 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:20.813542 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:20.814831 19941 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } attempt: 2 I0918 20:38:20.826939 19941 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } has no permanent_uuid. Determining permanent_uuid... W0918 20:38:20.828640 19941 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:44401: Network error: Client connection negotiation failed: client connection to 127.18.222.60:44401: connect: Connection refused (error 111) I0918 20:38:20.844038 19998 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } attempt: 1 W0918 20:38:20.846174 19998 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:44401: Network error: Client connection negotiation failed: client connection to 127.18.222.60:44401: connect: Connection refused (error 111) I0918 20:38:20.857046 19941 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } attempt: 1 W0918 20:38:20.858958 19941 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:44401: Network error: Client connection negotiation failed: client connection to 127.18.222.60:44401: connect: Connection refused (error 111) I0918 20:38:20.889858 19998 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } attempt: 2 W0918 20:38:20.891927 19998 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:44401: Network error: Client connection negotiation failed: client connection to 127.18.222.60:44401: connect: Connection refused (error 111) I0918 20:38:20.901579 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:44401 I0918 20:38:20.903646 19320 webserver.cc:374] Webserver started at http://127.18.222.60:35067/ using document root and password file I0918 20:38:20.904863 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0918 20:38:20.905795 20055 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:20.909344 19941 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } attempt: 2 I0918 20:38:20.910037 20055 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:20.923460 20055 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:20.926573 19941 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad: Bootstrap starting. I0918 20:38:20.929431 19941 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad: No blocks or log segments found. Creating new log. I0918 20:38:20.933374 19941 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad: No bootstrap required, opened a new log I0918 20:38:20.933336 20055 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:20.935528 19941 raft_consensus.cc:340] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } I0918 20:38:20.936110 19941 raft_consensus.cc:366] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:20.936298 19941 raft_consensus.cc:705] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa8ea8cc088448b988deacd9000a0ad, State: Initialized, Role: FOLLOWER I0918 20:38:20.936854 19941 consensus_queue.cc:229] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } I0918 20:38:20.938560 19941 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } } I0918 20:38:20.939352 19941 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:20.938871 20059 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } } I0918 20:38:20.939779 20059 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:20.940551 19941 sys_catalog.cc:428] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad [sys.catalog]: configured and running, proceeding with master startup. I0918 20:38:20.941730 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 W0918 20:38:20.944351 20060 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:38:20.944562 20060 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:20.947360 20055 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03: Bootstrap starting. I0918 20:38:20.950064 20055 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03: No blocks or log segments found. Creating new log. I0918 20:38:20.953112 20055 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03: No bootstrap required, opened a new log I0918 20:38:20.954546 20055 raft_consensus.cc:340] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } I0918 20:38:20.955044 20055 raft_consensus.cc:366] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:20.955212 20055 raft_consensus.cc:705] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252ccf1ea83e476392b25d8272d26a03, State: Initialized, Role: FOLLOWER I0918 20:38:20.955675 20055 consensus_queue.cc:229] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } I0918 20:38:20.957314 20055 sys_catalog.cc:340] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } } I0918 20:38:20.958053 20055 sys_catalog.cc:343] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:20.957599 20062 sys_catalog.cc:340] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } } I0918 20:38:20.958403 20062 sys_catalog.cc:343] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:20.959112 20055 sys_catalog.cc:428] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [sys.catalog]: configured and running, proceeding with master startup. W0918 20:38:20.962231 20063 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:38:20.962432 20063 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:20.964359 19998 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } attempt: 3 I0918 20:38:20.977843 19998 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b: Bootstrap starting. I0918 20:38:20.980666 19998 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b: No blocks or log segments found. Creating new log. I0918 20:38:20.983803 19998 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b: No bootstrap required, opened a new log I0918 20:38:20.985340 19998 raft_consensus.cc:340] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } I0918 20:38:20.985832 19998 raft_consensus.cc:366] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:20.986033 19998 raft_consensus.cc:705] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ee39e0fecc41c0bb51dd5ccac4848b, State: Initialized, Role: FOLLOWER I0918 20:38:20.986575 19998 consensus_queue.cc:229] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } I0918 20:38:20.988217 19998 sys_catalog.cc:340] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } } I0918 20:38:20.988737 19998 sys_catalog.cc:343] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:20.988459 20065 sys_catalog.cc:340] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } } I0918 20:38:20.989050 20065 sys_catalog.cc:343] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:20.989655 19998 sys_catalog.cc:428] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b [sys.catalog]: configured and running, proceeding with master startup. I0918 20:38:20.990564 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 W0918 20:38:20.992539 20066 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:38:20.992766 20066 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:21.005542 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 128092us I0918 20:38:21.005919 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:21.006086 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:21.006307 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:21.006428 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:21.007531 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/ts-0-root/instance: uuid: "3810d972c23b409788635c10f7594962" format_stamp: "Formatted at 2019-09-18 20:38:21 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:21.017782 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.008s sys 0.001s I0918 20:38:21.019281 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:21.027123 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:21.027325 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/ts-0-root uuid: "3810d972c23b409788635c10f7594962" format_stamp: "Formatted at 2019-09-18 20:38:21 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:21.027587 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:21.047456 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:21.047623 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:21.083436 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:39537 I0918 20:38:21.085238 19320 webserver.cc:374] Webserver started at http://127.18.222.1:33789/ using document root and password file I0918 20:38:21.106724 20131 heartbeater.cc:322] Connected to a master server at 127.18.222.60:44401 I0918 20:38:21.107085 20131 heartbeater.cc:406] Registering TS with master... I0918 20:38:21.108155 20131 heartbeater.cc:455] Master 127.18.222.60:44401 requested a full tablet report, sending... I0918 20:38:21.111083 20022 ts_manager.cc:186] Registered new tserver with Master: 3810d972c23b409788635c10f7594962 (127.18.222.1:39537) I0918 20:38:21.113250 20132 heartbeater.cc:322] Connected to a master server at 127.18.222.61:45425 I0918 20:38:21.113534 20132 heartbeater.cc:406] Registering TS with master... I0918 20:38:21.114013 20132 heartbeater.cc:455] Master 127.18.222.61:45425 requested a full tablet report, sending... I0918 20:38:21.114327 20134 heartbeater.cc:322] Connected to a master server at 127.18.222.62:33587 I0918 20:38:21.114805 20134 heartbeater.cc:406] Registering TS with master... I0918 20:38:21.115432 20134 heartbeater.cc:455] Master 127.18.222.62:33587 requested a full tablet report, sending... I0918 20:38:21.116523 19965 ts_manager.cc:186] Registered new tserver with Master: 3810d972c23b409788635c10f7594962 (127.18.222.1:39537) I0918 20:38:21.116964 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 128092us I0918 20:38:21.117363 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:21.117584 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:21.117805 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:21.117933 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:21.117923 19908 ts_manager.cc:186] Registered new tserver with Master: 3810d972c23b409788635c10f7594962 (127.18.222.1:39537) I0918 20:38:21.119410 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/ts-1-root/instance: uuid: "252cea633d73492e8b9fb46364ebe78f" format_stamp: "Formatted at 2019-09-18 20:38:21 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:21.129731 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.009s sys 0.000s I0918 20:38:21.132390 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:21.140817 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I0918 20:38:21.141047 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/ts-1-root uuid: "252cea633d73492e8b9fb46364ebe78f" format_stamp: "Formatted at 2019-09-18 20:38:21 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:21.141297 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:21.179858 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:21.180073 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:21.226414 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.2:42709 I0918 20:38:21.229480 19320 webserver.cc:374] Webserver started at http://127.18.222.2:45239/ using document root and password file I0918 20:38:21.244241 20062 raft_consensus.cc:472] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:21.244766 20062 raft_consensus.cc:494] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } I0918 20:38:21.246461 20062 leader_election.cc:296] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fa8ea8cc088448b988deacd9000a0ad (127.18.222.62:33587), 35ee39e0fecc41c0bb51dd5ccac4848b (127.18.222.61:45425) I0918 20:38:21.247849 19975 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "252ccf1ea83e476392b25d8272d26a03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" is_pre_election: true I0918 20:38:21.248865 19975 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 252ccf1ea83e476392b25d8272d26a03 in term 0. I0918 20:38:21.250267 20006 leader_election.cc:310] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 252ccf1ea83e476392b25d8272d26a03, 35ee39e0fecc41c0bb51dd5ccac4848b; no voters: I0918 20:38:21.251308 20062 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:21.248131 19918 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "252ccf1ea83e476392b25d8272d26a03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa8ea8cc088448b988deacd9000a0ad" is_pre_election: true I0918 20:38:21.252090 19918 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 252ccf1ea83e476392b25d8272d26a03 in term 0. I0918 20:38:21.251600 20062 raft_consensus.cc:472] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:21.253005 20062 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:21.258538 20062 raft_consensus.cc:494] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } I0918 20:38:21.260401 20062 leader_election.cc:296] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [CANDIDATE]: Term 1 election: Requested vote from peers 5fa8ea8cc088448b988deacd9000a0ad (127.18.222.62:33587), 35ee39e0fecc41c0bb51dd5ccac4848b (127.18.222.61:45425) I0918 20:38:21.263239 19918 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "252ccf1ea83e476392b25d8272d26a03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa8ea8cc088448b988deacd9000a0ad" I0918 20:38:21.264231 19918 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:21.265272 20203 heartbeater.cc:322] Connected to a master server at 127.18.222.61:45425 I0918 20:38:21.265588 20203 heartbeater.cc:406] Registering TS with master... I0918 20:38:21.266201 20203 heartbeater.cc:455] Master 127.18.222.61:45425 requested a full tablet report, sending... I0918 20:38:21.264039 19975 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "252ccf1ea83e476392b25d8272d26a03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" I0918 20:38:21.268590 19975 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:21.270364 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 128092us I0918 20:38:21.270587 20202 heartbeater.cc:322] Connected to a master server at 127.18.222.60:44401 I0918 20:38:21.270874 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:21.270954 20202 heartbeater.cc:406] Registering TS with master... I0918 20:38:21.271040 20204 heartbeater.cc:322] Connected to a master server at 127.18.222.62:33587 I0918 20:38:21.271127 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:21.268673 19964 ts_manager.cc:186] Registered new tserver with Master: 252cea633d73492e8b9fb46364ebe78f (127.18.222.2:42709) I0918 20:38:21.271621 20204 heartbeater.cc:406] Registering TS with master... I0918 20:38:21.271760 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:21.271917 20202 heartbeater.cc:455] Master 127.18.222.60:44401 requested a full tablet report, sending... I0918 20:38:21.272488 20204 heartbeater.cc:455] Master 127.18.222.62:33587 requested a full tablet report, sending... I0918 20:38:21.272814 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:21.274160 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/ts-2-root/instance: uuid: "60645d4f7df445de88c17b9c461ebcc4" format_stamp: "Formatted at 2019-09-18 20:38:21 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:21.274799 20022 ts_manager.cc:186] Registered new tserver with Master: 252cea633d73492e8b9fb46364ebe78f (127.18.222.2:42709) I0918 20:38:21.278172 19975 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 252ccf1ea83e476392b25d8272d26a03 in term 1. I0918 20:38:21.279346 20006 leader_election.cc:310] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 252ccf1ea83e476392b25d8272d26a03, 35ee39e0fecc41c0bb51dd5ccac4848b; no voters: I0918 20:38:21.280362 20062 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:21.281874 19918 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 252ccf1ea83e476392b25d8272d26a03 in term 1. I0918 20:38:21.283565 19908 ts_manager.cc:186] Registered new tserver with Master: 252cea633d73492e8b9fb46364ebe78f (127.18.222.2:42709) I0918 20:38:21.286417 20062 raft_consensus.cc:667] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [term 1 LEADER]: Becoming Leader. State: Replica: 252ccf1ea83e476392b25d8272d26a03, State: Running, Role: LEADER I0918 20:38:21.287277 20062 consensus_queue.cc:206] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } I0918 20:38:21.292383 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.018s user 0.016s sys 0.000s I0918 20:38:21.294457 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:21.295841 20209 sys_catalog.cc:340] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 252ccf1ea83e476392b25d8272d26a03. Latest consensus state: current_term: 1 leader_uuid: "252ccf1ea83e476392b25d8272d26a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } } I0918 20:38:21.296661 20209 sys_catalog.cc:343] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [sys.catalog]: This master's current role is: LEADER I0918 20:38:21.301975 20212 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:38:21.308720 20212 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:38:21.321215 19320 fs_manager.cc:419] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s I0918 20:38:21.321636 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/ts-2-root uuid: "60645d4f7df445de88c17b9c461ebcc4" format_stamp: "Formatted at 2019-09-18 20:38:21 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:21.322063 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1568839089958813-19320/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:21.391880 19975 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b [term 1 FOLLOWER]: Refusing update from remote peer 252ccf1ea83e476392b25d8272d26a03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:21.393450 20062 consensus_queue.cc:984] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:21.394404 19918 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad [term 1 FOLLOWER]: Refusing update from remote peer 252ccf1ea83e476392b25d8272d26a03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:21.396471 20062 consensus_queue.cc:984] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:21.411728 20065 sys_catalog.cc:340] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 252ccf1ea83e476392b25d8272d26a03. Latest consensus state: current_term: 1 leader_uuid: "252ccf1ea83e476392b25d8272d26a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } } I0918 20:38:21.413305 20065 sys_catalog.cc:343] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:21.415387 20059 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 252ccf1ea83e476392b25d8272d26a03. Latest consensus state: current_term: 1 leader_uuid: "252ccf1ea83e476392b25d8272d26a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } } I0918 20:38:21.416394 20059 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:21.436913 20062 sys_catalog.cc:340] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "252ccf1ea83e476392b25d8272d26a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } } I0918 20:38:21.437934 20062 sys_catalog.cc:343] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [sys.catalog]: This master's current role is: LEADER I0918 20:38:21.440668 20065 sys_catalog.cc:340] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "252ccf1ea83e476392b25d8272d26a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } } I0918 20:38:21.441215 20065 sys_catalog.cc:343] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:21.448657 20212 catalog_manager.cc:981] Generated new certificate authority record I0918 20:38:21.448932 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:21.449187 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:21.452234 20212 catalog_manager.cc:1106] Loading token signing keys... I0918 20:38:21.452514 20209 sys_catalog.cc:340] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "252ccf1ea83e476392b25d8272d26a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } } I0918 20:38:21.460012 20209 sys_catalog.cc:343] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [sys.catalog]: This master's current role is: LEADER I0918 20:38:21.467155 20059 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "252ccf1ea83e476392b25d8272d26a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa8ea8cc088448b988deacd9000a0ad" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 33587 } } peers { permanent_uuid: "35ee39e0fecc41c0bb51dd5ccac4848b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45425 } } peers { permanent_uuid: "252ccf1ea83e476392b25d8272d26a03" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44401 } } } I0918 20:38:21.470041 20059 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:21.489533 20212 catalog_manager.cc:4446] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03: Generated new TSK 0 I0918 20:38:21.490389 20212 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:38:21.502410 20022 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54590 I0918 20:38:21.504006 20021 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54596 I0918 20:38:21.515455 20131 heartbeater.cc:447] Master 127.18.222.60:44401 was elected leader, sending a full tablet report... I0918 20:38:21.518898 20202 heartbeater.cc:447] Master 127.18.222.60:44401 was elected leader, sending a full tablet report... I0918 20:38:21.556679 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.3:43027 I0918 20:38:21.563252 19320 webserver.cc:374] Webserver started at http://127.18.222.3:39175/ using document root and password file I0918 20:38:21.600570 20285 heartbeater.cc:322] Connected to a master server at 127.18.222.60:44401 I0918 20:38:21.600936 20285 heartbeater.cc:406] Registering TS with master... I0918 20:38:21.601579 20285 heartbeater.cc:455] Master 127.18.222.60:44401 requested a full tablet report, sending... I0918 20:38:21.602701 20286 heartbeater.cc:322] Connected to a master server at 127.18.222.61:45425 I0918 20:38:21.602982 20286 heartbeater.cc:406] Registering TS with master... I0918 20:38:21.603581 20286 heartbeater.cc:455] Master 127.18.222.61:45425 requested a full tablet report, sending... I0918 20:38:21.604526 20021 ts_manager.cc:186] Registered new tserver with Master: 60645d4f7df445de88c17b9c461ebcc4 (127.18.222.3:43027) I0918 20:38:21.607298 20021 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54602 I0918 20:38:21.607875 19964 ts_manager.cc:186] Registered new tserver with Master: 60645d4f7df445de88c17b9c461ebcc4 (127.18.222.3:43027) I0918 20:38:21.611901 20288 heartbeater.cc:322] Connected to a master server at 127.18.222.62:33587 I0918 20:38:21.612244 20288 heartbeater.cc:406] Registering TS with master... I0918 20:38:21.612545 20288 heartbeater.cc:455] Master 127.18.222.62:33587 requested a full tablet report, sending... I0918 20:38:21.614830 19907 ts_manager.cc:186] Registered new tserver with Master: 60645d4f7df445de88c17b9c461ebcc4 (127.18.222.3:43027) I0918 20:38:21.616024 19320 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.030577192s I0918 20:38:21.621058 20285 heartbeater.cc:447] Master 127.18.222.60:44401 was elected leader, sending a full tablet report... I0918 20:38:21.660128 20022 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54612: name: "ksck-test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\n\000\000\000\004\001\024\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:38:21.662170 20022 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:38:21.769418 20252 tablet_service.cc:1022] Processing CreateTablet for tablet 319bc51fcd494e1b92a4837dcd658383 (table=ksck-test-table [id=78a7f423710f4ec3971df9be7d81e3de]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:21.770582 20252 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 319bc51fcd494e1b92a4837dcd658383. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:21.794435 20251 tablet_service.cc:1022] Processing CreateTablet for tablet e46166b371974ac8b3f2bb853951c7a6 (table=ksck-test-table [id=78a7f423710f4ec3971df9be7d81e3de]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:21.795680 20251 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet e46166b371974ac8b3f2bb853951c7a6. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:21.801290 20250 tablet_service.cc:1022] Processing CreateTablet for tablet 052ace92eef0481cae05a81aec482e05 (table=ksck-test-table [id=78a7f423710f4ec3971df9be7d81e3de]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:21.802526 20250 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 052ace92eef0481cae05a81aec482e05. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:21.846351 20306 tablet_bootstrap.cc:439] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4: Bootstrap starting. I0918 20:38:21.851920 20306 tablet_bootstrap.cc:590] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4: No blocks or log segments found. Creating new log. I0918 20:38:21.852459 20098 tablet_service.cc:1022] Processing CreateTablet for tablet 319bc51fcd494e1b92a4837dcd658383 (table=ksck-test-table [id=78a7f423710f4ec3971df9be7d81e3de]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:21.853487 20098 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 319bc51fcd494e1b92a4837dcd658383. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:21.856355 20097 tablet_service.cc:1022] Processing CreateTablet for tablet e46166b371974ac8b3f2bb853951c7a6 (table=ksck-test-table [id=78a7f423710f4ec3971df9be7d81e3de]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:21.857385 20097 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet e46166b371974ac8b3f2bb853951c7a6. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:21.866701 20096 tablet_service.cc:1022] Processing CreateTablet for tablet 052ace92eef0481cae05a81aec482e05 (table=ksck-test-table [id=78a7f423710f4ec3971df9be7d81e3de]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:21.867178 20169 tablet_service.cc:1022] Processing CreateTablet for tablet 319bc51fcd494e1b92a4837dcd658383 (table=ksck-test-table [id=78a7f423710f4ec3971df9be7d81e3de]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:21.867799 20096 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 052ace92eef0481cae05a81aec482e05. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:21.868654 20167 tablet_service.cc:1022] Processing CreateTablet for tablet 052ace92eef0481cae05a81aec482e05 (table=ksck-test-table [id=78a7f423710f4ec3971df9be7d81e3de]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:21.869721 20167 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 052ace92eef0481cae05a81aec482e05. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:21.872979 20169 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 319bc51fcd494e1b92a4837dcd658383. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:21.872221 20168 tablet_service.cc:1022] Processing CreateTablet for tablet e46166b371974ac8b3f2bb853951c7a6 (table=ksck-test-table [id=78a7f423710f4ec3971df9be7d81e3de]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:21.874969 20168 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet e46166b371974ac8b3f2bb853951c7a6. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:21.910997 20308 tablet_bootstrap.cc:439] T e46166b371974ac8b3f2bb853951c7a6 P 3810d972c23b409788635c10f7594962: Bootstrap starting. I0918 20:38:21.916754 20306 tablet_bootstrap.cc:439] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4: No bootstrap required, opened a new log I0918 20:38:21.920214 20306 ts_tablet_manager.cc:1085] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4: Time spent bootstrapping tablet: real 0.074s user 0.011s sys 0.016s I0918 20:38:21.923218 20306 raft_consensus.cc:340] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } I0918 20:38:21.924201 20306 raft_consensus.cc:366] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:21.924487 20306 raft_consensus.cc:705] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60645d4f7df445de88c17b9c461ebcc4, State: Initialized, Role: FOLLOWER I0918 20:38:21.925288 20306 consensus_queue.cc:229] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } I0918 20:38:21.931260 20308 tablet_bootstrap.cc:590] T e46166b371974ac8b3f2bb853951c7a6 P 3810d972c23b409788635c10f7594962: No blocks or log segments found. Creating new log. I0918 20:38:21.939249 20306 ts_tablet_manager.cc:1113] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4: Time spent starting tablet: real 0.019s user 0.009s sys 0.001s I0918 20:38:21.940213 20306 tablet_bootstrap.cc:439] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4: Bootstrap starting. I0918 20:38:21.953492 20060 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad: acquiring CA information for follower catalog manager: success I0918 20:38:21.957432 20060 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:38:21.959522 20309 tablet_bootstrap.cc:439] T 319bc51fcd494e1b92a4837dcd658383 P 252cea633d73492e8b9fb46364ebe78f: Bootstrap starting. I0918 20:38:21.967108 20308 tablet_bootstrap.cc:439] T e46166b371974ac8b3f2bb853951c7a6 P 3810d972c23b409788635c10f7594962: No bootstrap required, opened a new log I0918 20:38:21.967648 20308 ts_tablet_manager.cc:1085] T e46166b371974ac8b3f2bb853951c7a6 P 3810d972c23b409788635c10f7594962: Time spent bootstrapping tablet: real 0.057s user 0.011s sys 0.015s I0918 20:38:21.970074 20308 raft_consensus.cc:340] T e46166b371974ac8b3f2bb853951c7a6 P 3810d972c23b409788635c10f7594962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } I0918 20:38:21.970921 20308 raft_consensus.cc:366] T e46166b371974ac8b3f2bb853951c7a6 P 3810d972c23b409788635c10f7594962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:21.975818 20308 raft_consensus.cc:705] T e46166b371974ac8b3f2bb853951c7a6 P 3810d972c23b409788635c10f7594962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3810d972c23b409788635c10f7594962, State: Initialized, Role: FOLLOWER I0918 20:38:21.976466 20308 consensus_queue.cc:229] T e46166b371974ac8b3f2bb853951c7a6 P 3810d972c23b409788635c10f7594962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } I0918 20:38:21.964612 20306 tablet_bootstrap.cc:590] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4: No blocks or log segments found. Creating new log. I0918 20:38:21.984629 20309 tablet_bootstrap.cc:590] T 319bc51fcd494e1b92a4837dcd658383 P 252cea633d73492e8b9fb46364ebe78f: No blocks or log segments found. Creating new log. I0918 20:38:21.985437 20308 ts_tablet_manager.cc:1113] T e46166b371974ac8b3f2bb853951c7a6 P 3810d972c23b409788635c10f7594962: Time spent starting tablet: real 0.017s user 0.004s sys 0.002s I0918 20:38:21.986265 20308 tablet_bootstrap.cc:439] T 052ace92eef0481cae05a81aec482e05 P 3810d972c23b409788635c10f7594962: Bootstrap starting. I0918 20:38:21.988399 20306 tablet_bootstrap.cc:439] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4: No bootstrap required, opened a new log I0918 20:38:21.988939 20306 ts_tablet_manager.cc:1085] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.001s I0918 20:38:21.990058 20308 tablet_bootstrap.cc:590] T 052ace92eef0481cae05a81aec482e05 P 3810d972c23b409788635c10f7594962: No blocks or log segments found. Creating new log. I0918 20:38:21.992563 20306 raft_consensus.cc:340] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } I0918 20:38:21.993468 20306 raft_consensus.cc:366] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:21.993758 20306 raft_consensus.cc:705] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60645d4f7df445de88c17b9c461ebcc4, State: Initialized, Role: FOLLOWER I0918 20:38:22.007225 20066 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b: acquiring CA information for follower catalog manager: success I0918 20:38:21.994627 20306 consensus_queue.cc:229] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } I0918 20:38:22.017333 20308 tablet_bootstrap.cc:439] T 052ace92eef0481cae05a81aec482e05 P 3810d972c23b409788635c10f7594962: No bootstrap required, opened a new log I0918 20:38:22.023401 20308 ts_tablet_manager.cc:1085] T 052ace92eef0481cae05a81aec482e05 P 3810d972c23b409788635c10f7594962: Time spent bootstrapping tablet: real 0.037s user 0.015s sys 0.010s I0918 20:38:22.022861 20309 tablet_bootstrap.cc:439] T 319bc51fcd494e1b92a4837dcd658383 P 252cea633d73492e8b9fb46364ebe78f: No bootstrap required, opened a new log I0918 20:38:22.021095 20306 ts_tablet_manager.cc:1113] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4: Time spent starting tablet: real 0.032s user 0.003s sys 0.002s I0918 20:38:22.024821 20306 tablet_bootstrap.cc:439] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4: Bootstrap starting. I0918 20:38:22.027850 20309 ts_tablet_manager.cc:1085] T 319bc51fcd494e1b92a4837dcd658383 P 252cea633d73492e8b9fb46364ebe78f: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.006s I0918 20:38:22.031296 20066 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:38:22.032193 20309 raft_consensus.cc:340] T 319bc51fcd494e1b92a4837dcd658383 P 252cea633d73492e8b9fb46364ebe78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } I0918 20:38:22.032712 20309 raft_consensus.cc:366] T 319bc51fcd494e1b92a4837dcd658383 P 252cea633d73492e8b9fb46364ebe78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:22.032902 20309 raft_consensus.cc:705] T 319bc51fcd494e1b92a4837dcd658383 P 252cea633d73492e8b9fb46364ebe78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252cea633d73492e8b9fb46364ebe78f, State: Initialized, Role: FOLLOWER I0918 20:38:22.035748 20309 consensus_queue.cc:229] T 319bc51fcd494e1b92a4837dcd658383 P 252cea633d73492e8b9fb46364ebe78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } I0918 20:38:22.034972 20306 tablet_bootstrap.cc:590] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4: No blocks or log segments found. Creating new log. I0918 20:38:22.031297 20308 raft_consensus.cc:340] T 052ace92eef0481cae05a81aec482e05 P 3810d972c23b409788635c10f7594962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } I0918 20:38:22.038977 20309 ts_tablet_manager.cc:1113] T 319bc51fcd494e1b92a4837dcd658383 P 252cea633d73492e8b9fb46364ebe78f: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s I0918 20:38:22.039819 20308 raft_consensus.cc:366] T 052ace92eef0481cae05a81aec482e05 P 3810d972c23b409788635c10f7594962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:22.039885 20309 tablet_bootstrap.cc:439] T 052ace92eef0481cae05a81aec482e05 P 252cea633d73492e8b9fb46364ebe78f: Bootstrap starting. I0918 20:38:22.040205 20308 raft_consensus.cc:705] T 052ace92eef0481cae05a81aec482e05 P 3810d972c23b409788635c10f7594962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3810d972c23b409788635c10f7594962, State: Initialized, Role: FOLLOWER I0918 20:38:22.041067 20308 consensus_queue.cc:229] T 052ace92eef0481cae05a81aec482e05 P 3810d972c23b409788635c10f7594962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } I0918 20:38:22.043401 20308 ts_tablet_manager.cc:1113] T 052ace92eef0481cae05a81aec482e05 P 3810d972c23b409788635c10f7594962: Time spent starting tablet: real 0.020s user 0.002s sys 0.003s I0918 20:38:22.044224 20308 tablet_bootstrap.cc:439] T 319bc51fcd494e1b92a4837dcd658383 P 3810d972c23b409788635c10f7594962: Bootstrap starting. I0918 20:38:22.045424 20309 tablet_bootstrap.cc:590] T 052ace92eef0481cae05a81aec482e05 P 252cea633d73492e8b9fb46364ebe78f: No blocks or log segments found. Creating new log. I0918 20:38:22.048322 20308 tablet_bootstrap.cc:590] T 319bc51fcd494e1b92a4837dcd658383 P 3810d972c23b409788635c10f7594962: No blocks or log segments found. Creating new log. I0918 20:38:22.069950 20306 tablet_bootstrap.cc:439] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4: No bootstrap required, opened a new log I0918 20:38:22.069272 20308 tablet_bootstrap.cc:439] T 319bc51fcd494e1b92a4837dcd658383 P 3810d972c23b409788635c10f7594962: No bootstrap required, opened a new log I0918 20:38:22.071133 20308 ts_tablet_manager.cc:1085] T 319bc51fcd494e1b92a4837dcd658383 P 3810d972c23b409788635c10f7594962: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.009s I0918 20:38:22.072140 20309 tablet_bootstrap.cc:439] T 052ace92eef0481cae05a81aec482e05 P 252cea633d73492e8b9fb46364ebe78f: No bootstrap required, opened a new log I0918 20:38:22.072646 20309 ts_tablet_manager.cc:1085] T 052ace92eef0481cae05a81aec482e05 P 252cea633d73492e8b9fb46364ebe78f: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.005s I0918 20:38:22.073374 20308 raft_consensus.cc:340] T 319bc51fcd494e1b92a4837dcd658383 P 3810d972c23b409788635c10f7594962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } I0918 20:38:22.074134 20308 raft_consensus.cc:366] T 319bc51fcd494e1b92a4837dcd658383 P 3810d972c23b409788635c10f7594962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:22.074389 20308 raft_consensus.cc:705] T 319bc51fcd494e1b92a4837dcd658383 P 3810d972c23b409788635c10f7594962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3810d972c23b409788635c10f7594962, State: Initialized, Role: FOLLOWER I0918 20:38:22.070770 20306 ts_tablet_manager.cc:1085] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.017s I0918 20:38:22.076608 20306 raft_consensus.cc:340] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } I0918 20:38:22.077405 20306 raft_consensus.cc:366] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:22.077684 20306 raft_consensus.cc:705] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60645d4f7df445de88c17b9c461ebcc4, State: Initialized, Role: FOLLOWER I0918 20:38:22.079227 20306 consensus_queue.cc:229] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } I0918 20:38:22.074591 20309 raft_consensus.cc:340] T 052ace92eef0481cae05a81aec482e05 P 252cea633d73492e8b9fb46364ebe78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } I0918 20:38:22.080451 20309 raft_consensus.cc:366] T 052ace92eef0481cae05a81aec482e05 P 252cea633d73492e8b9fb46364ebe78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:22.080786 20309 raft_consensus.cc:705] T 052ace92eef0481cae05a81aec482e05 P 252cea633d73492e8b9fb46364ebe78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252cea633d73492e8b9fb46364ebe78f, State: Initialized, Role: FOLLOWER I0918 20:38:22.081387 20309 consensus_queue.cc:229] T 052ace92eef0481cae05a81aec482e05 P 252cea633d73492e8b9fb46364ebe78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } I0918 20:38:22.084437 20309 ts_tablet_manager.cc:1113] T 052ace92eef0481cae05a81aec482e05 P 252cea633d73492e8b9fb46364ebe78f: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s I0918 20:38:22.085222 20309 tablet_bootstrap.cc:439] T e46166b371974ac8b3f2bb853951c7a6 P 252cea633d73492e8b9fb46364ebe78f: Bootstrap starting. I0918 20:38:22.086200 20306 ts_tablet_manager.cc:1113] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s I0918 20:38:22.086984 20308 consensus_queue.cc:229] T 319bc51fcd494e1b92a4837dcd658383 P 3810d972c23b409788635c10f7594962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } I0918 20:38:22.088814 20309 tablet_bootstrap.cc:590] T e46166b371974ac8b3f2bb853951c7a6 P 252cea633d73492e8b9fb46364ebe78f: No blocks or log segments found. Creating new log. I0918 20:38:22.090322 20308 ts_tablet_manager.cc:1113] T 319bc51fcd494e1b92a4837dcd658383 P 3810d972c23b409788635c10f7594962: Time spent starting tablet: real 0.019s user 0.000s sys 0.005s I0918 20:38:22.113435 20310 raft_consensus.cc:472] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:22.113688 20309 tablet_bootstrap.cc:439] T e46166b371974ac8b3f2bb853951c7a6 P 252cea633d73492e8b9fb46364ebe78f: No bootstrap required, opened a new log I0918 20:38:22.113950 20310 raft_consensus.cc:494] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } I0918 20:38:22.114681 20309 ts_tablet_manager.cc:1085] T e46166b371974ac8b3f2bb853951c7a6 P 252cea633d73492e8b9fb46364ebe78f: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.007s I0918 20:38:22.116971 20310 leader_election.cc:296] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3810d972c23b409788635c10f7594962 (127.18.222.1:39537), 252cea633d73492e8b9fb46364ebe78f (127.18.222.2:42709) I0918 20:38:22.117169 20309 raft_consensus.cc:340] T e46166b371974ac8b3f2bb853951c7a6 P 252cea633d73492e8b9fb46364ebe78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } I0918 20:38:22.118510 20309 raft_consensus.cc:366] T e46166b371974ac8b3f2bb853951c7a6 P 252cea633d73492e8b9fb46364ebe78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:22.118731 20309 raft_consensus.cc:705] T e46166b371974ac8b3f2bb853951c7a6 P 252cea633d73492e8b9fb46364ebe78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252cea633d73492e8b9fb46364ebe78f, State: Initialized, Role: FOLLOWER I0918 20:38:22.119805 20309 consensus_queue.cc:229] T e46166b371974ac8b3f2bb853951c7a6 P 252cea633d73492e8b9fb46364ebe78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } I0918 20:38:22.132632 20309 ts_tablet_manager.cc:1113] T e46166b371974ac8b3f2bb853951c7a6 P 252cea633d73492e8b9fb46364ebe78f: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s I0918 20:38:22.153950 20108 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e46166b371974ac8b3f2bb853951c7a6" candidate_uuid: "60645d4f7df445de88c17b9c461ebcc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3810d972c23b409788635c10f7594962" is_pre_election: true I0918 20:38:22.154709 20108 raft_consensus.cc:2350] T e46166b371974ac8b3f2bb853951c7a6 P 3810d972c23b409788635c10f7594962 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60645d4f7df445de88c17b9c461ebcc4 in term 0. I0918 20:38:22.156100 20216 leader_election.cc:310] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3810d972c23b409788635c10f7594962, 60645d4f7df445de88c17b9c461ebcc4; no voters: I0918 20:38:22.156406 20179 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e46166b371974ac8b3f2bb853951c7a6" candidate_uuid: "60645d4f7df445de88c17b9c461ebcc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252cea633d73492e8b9fb46364ebe78f" is_pre_election: true I0918 20:38:22.157132 20179 raft_consensus.cc:2350] T e46166b371974ac8b3f2bb853951c7a6 P 252cea633d73492e8b9fb46364ebe78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60645d4f7df445de88c17b9c461ebcc4 in term 0. I0918 20:38:22.157831 20310 raft_consensus.cc:2644] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:22.158179 20310 raft_consensus.cc:472] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:22.158507 20310 raft_consensus.cc:2890] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:22.165535 20310 raft_consensus.cc:494] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } I0918 20:38:22.175472 20310 leader_election.cc:296] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 [CANDIDATE]: Term 1 election: Requested vote from peers 3810d972c23b409788635c10f7594962 (127.18.222.1:39537), 252cea633d73492e8b9fb46364ebe78f (127.18.222.2:42709) I0918 20:38:22.181933 20108 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e46166b371974ac8b3f2bb853951c7a6" candidate_uuid: "60645d4f7df445de88c17b9c461ebcc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3810d972c23b409788635c10f7594962" I0918 20:38:22.182695 20108 raft_consensus.cc:2890] T e46166b371974ac8b3f2bb853951c7a6 P 3810d972c23b409788635c10f7594962 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:22.183296 20179 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e46166b371974ac8b3f2bb853951c7a6" candidate_uuid: "60645d4f7df445de88c17b9c461ebcc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252cea633d73492e8b9fb46364ebe78f" I0918 20:38:22.183883 20179 raft_consensus.cc:2890] T e46166b371974ac8b3f2bb853951c7a6 P 252cea633d73492e8b9fb46364ebe78f [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:22.188746 20108 raft_consensus.cc:2350] T e46166b371974ac8b3f2bb853951c7a6 P 3810d972c23b409788635c10f7594962 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60645d4f7df445de88c17b9c461ebcc4 in term 1. I0918 20:38:22.189087 20179 raft_consensus.cc:2350] T e46166b371974ac8b3f2bb853951c7a6 P 252cea633d73492e8b9fb46364ebe78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60645d4f7df445de88c17b9c461ebcc4 in term 1. I0918 20:38:22.189857 20216 leader_election.cc:310] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3810d972c23b409788635c10f7594962, 60645d4f7df445de88c17b9c461ebcc4; no voters: I0918 20:38:22.191164 20310 raft_consensus.cc:2644] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:22.199106 20310 raft_consensus.cc:667] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 [term 1 LEADER]: Becoming Leader. State: Replica: 60645d4f7df445de88c17b9c461ebcc4, State: Running, Role: LEADER I0918 20:38:22.199954 20310 consensus_queue.cc:206] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } I0918 20:38:22.212674 20022 catalog_manager.cc:4131] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 reported cstate change: term changed from 0 to 1, leader changed from to 60645d4f7df445de88c17b9c461ebcc4 (127.18.222.3). New cstate: current_term: 1 leader_uuid: "60645d4f7df445de88c17b9c461ebcc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } health_report { overall_health: HEALTHY } } } I0918 20:38:22.244947 20310 raft_consensus.cc:472] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:22.245523 20310 raft_consensus.cc:494] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } I0918 20:38:22.247606 20323 raft_consensus.cc:472] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:22.248107 20323 raft_consensus.cc:494] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } I0918 20:38:22.252269 20310 leader_election.cc:296] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3810d972c23b409788635c10f7594962 (127.18.222.1:39537), 252cea633d73492e8b9fb46364ebe78f (127.18.222.2:42709) I0918 20:38:22.252542 20323 leader_election.cc:296] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3810d972c23b409788635c10f7594962 (127.18.222.1:39537), 252cea633d73492e8b9fb46364ebe78f (127.18.222.2:42709) I0918 20:38:22.253413 20108 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "052ace92eef0481cae05a81aec482e05" candidate_uuid: "60645d4f7df445de88c17b9c461ebcc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3810d972c23b409788635c10f7594962" is_pre_election: true I0918 20:38:22.253787 20179 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "052ace92eef0481cae05a81aec482e05" candidate_uuid: "60645d4f7df445de88c17b9c461ebcc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252cea633d73492e8b9fb46364ebe78f" is_pre_election: true I0918 20:38:22.254159 20108 raft_consensus.cc:2350] T 052ace92eef0481cae05a81aec482e05 P 3810d972c23b409788635c10f7594962 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60645d4f7df445de88c17b9c461ebcc4 in term 0. I0918 20:38:22.254030 20107 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "319bc51fcd494e1b92a4837dcd658383" candidate_uuid: "60645d4f7df445de88c17b9c461ebcc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3810d972c23b409788635c10f7594962" is_pre_election: true I0918 20:38:22.254451 20179 raft_consensus.cc:2350] T 052ace92eef0481cae05a81aec482e05 P 252cea633d73492e8b9fb46364ebe78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60645d4f7df445de88c17b9c461ebcc4 in term 0. I0918 20:38:22.254892 20107 raft_consensus.cc:2350] T 319bc51fcd494e1b92a4837dcd658383 P 3810d972c23b409788635c10f7594962 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60645d4f7df445de88c17b9c461ebcc4 in term 0. I0918 20:38:22.257166 20216 leader_election.cc:310] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3810d972c23b409788635c10f7594962, 60645d4f7df445de88c17b9c461ebcc4; no voters: I0918 20:38:22.258121 20179 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "319bc51fcd494e1b92a4837dcd658383" candidate_uuid: "60645d4f7df445de88c17b9c461ebcc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252cea633d73492e8b9fb46364ebe78f" is_pre_election: true I0918 20:38:22.259984 20179 raft_consensus.cc:2350] T 319bc51fcd494e1b92a4837dcd658383 P 252cea633d73492e8b9fb46364ebe78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60645d4f7df445de88c17b9c461ebcc4 in term 0. I0918 20:38:22.260931 20216 leader_election.cc:310] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3810d972c23b409788635c10f7594962, 60645d4f7df445de88c17b9c461ebcc4; no voters: I0918 20:38:22.261824 20323 raft_consensus.cc:2644] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:22.262193 20323 raft_consensus.cc:472] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:22.262518 20323 raft_consensus.cc:2890] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:22.262472 20310 raft_consensus.cc:2644] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:22.267817 20310 raft_consensus.cc:472] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:22.268277 20310 raft_consensus.cc:2890] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:22.280786 20310 raft_consensus.cc:494] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } I0918 20:38:22.282974 20107 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "319bc51fcd494e1b92a4837dcd658383" candidate_uuid: "60645d4f7df445de88c17b9c461ebcc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3810d972c23b409788635c10f7594962" I0918 20:38:22.283529 20107 raft_consensus.cc:2890] T 319bc51fcd494e1b92a4837dcd658383 P 3810d972c23b409788635c10f7594962 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:22.276857 20323 raft_consensus.cc:494] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } I0918 20:38:22.288621 20310 leader_election.cc:296] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 [CANDIDATE]: Term 1 election: Requested vote from peers 3810d972c23b409788635c10f7594962 (127.18.222.1:39537), 252cea633d73492e8b9fb46364ebe78f (127.18.222.2:42709) I0918 20:38:22.288832 20179 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "319bc51fcd494e1b92a4837dcd658383" candidate_uuid: "60645d4f7df445de88c17b9c461ebcc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252cea633d73492e8b9fb46364ebe78f" I0918 20:38:22.292178 20179 raft_consensus.cc:2890] T 319bc51fcd494e1b92a4837dcd658383 P 252cea633d73492e8b9fb46364ebe78f [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:22.293582 20108 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "052ace92eef0481cae05a81aec482e05" candidate_uuid: "60645d4f7df445de88c17b9c461ebcc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3810d972c23b409788635c10f7594962" I0918 20:38:22.294198 20108 raft_consensus.cc:2890] T 052ace92eef0481cae05a81aec482e05 P 3810d972c23b409788635c10f7594962 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:22.294050 20178 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "052ace92eef0481cae05a81aec482e05" candidate_uuid: "60645d4f7df445de88c17b9c461ebcc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252cea633d73492e8b9fb46364ebe78f" I0918 20:38:22.294704 20178 raft_consensus.cc:2890] T 052ace92eef0481cae05a81aec482e05 P 252cea633d73492e8b9fb46364ebe78f [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:22.299221 20179 raft_consensus.cc:2350] T 319bc51fcd494e1b92a4837dcd658383 P 252cea633d73492e8b9fb46364ebe78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60645d4f7df445de88c17b9c461ebcc4 in term 1. I0918 20:38:22.301641 20178 raft_consensus.cc:2350] T 052ace92eef0481cae05a81aec482e05 P 252cea633d73492e8b9fb46364ebe78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60645d4f7df445de88c17b9c461ebcc4 in term 1. I0918 20:38:22.290335 20107 raft_consensus.cc:2350] T 319bc51fcd494e1b92a4837dcd658383 P 3810d972c23b409788635c10f7594962 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60645d4f7df445de88c17b9c461ebcc4 in term 1. I0918 20:38:22.303638 20108 raft_consensus.cc:2350] T 052ace92eef0481cae05a81aec482e05 P 3810d972c23b409788635c10f7594962 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60645d4f7df445de88c17b9c461ebcc4 in term 1. I0918 20:38:22.304877 20216 leader_election.cc:310] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 252cea633d73492e8b9fb46364ebe78f, 60645d4f7df445de88c17b9c461ebcc4; no voters: I0918 20:38:22.306381 20216 leader_election.cc:310] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3810d972c23b409788635c10f7594962, 60645d4f7df445de88c17b9c461ebcc4; no voters: I0918 20:38:22.308450 20333 raft_consensus.cc:2644] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:22.309208 20310 raft_consensus.cc:2644] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:22.312042 20323 leader_election.cc:296] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 [CANDIDATE]: Term 1 election: Requested vote from peers 3810d972c23b409788635c10f7594962 (127.18.222.1:39537), 252cea633d73492e8b9fb46364ebe78f (127.18.222.2:42709) I0918 20:38:22.312116 20333 raft_consensus.cc:667] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 [term 1 LEADER]: Becoming Leader. State: Replica: 60645d4f7df445de88c17b9c461ebcc4, State: Running, Role: LEADER I0918 20:38:22.313555 20333 consensus_queue.cc:206] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } I0918 20:38:22.314761 20310 raft_consensus.cc:667] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 [term 1 LEADER]: Becoming Leader. State: Replica: 60645d4f7df445de88c17b9c461ebcc4, State: Running, Role: LEADER I0918 20:38:22.315455 20310 consensus_queue.cc:206] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } } I0918 20:38:22.327677 20020 catalog_manager.cc:4131] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 reported cstate change: term changed from 0 to 1, leader changed from to 60645d4f7df445de88c17b9c461ebcc4 (127.18.222.3). New cstate: current_term: 1 leader_uuid: "60645d4f7df445de88c17b9c461ebcc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } health_report { overall_health: UNKNOWN } } } I0918 20:38:22.330157 20020 catalog_manager.cc:4131] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 reported cstate change: term changed from 0 to 1, leader changed from to 60645d4f7df445de88c17b9c461ebcc4 (127.18.222.3). New cstate: current_term: 1 leader_uuid: "60645d4f7df445de88c17b9c461ebcc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60645d4f7df445de88c17b9c461ebcc4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43027 } health_report { overall_health: HEALTHY } } } I0918 20:38:22.689517 20333 consensus_queue.cc:984] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:22.737349 20310 consensus_queue.cc:984] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:22.781795 20310 consensus_queue.cc:984] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:22.815865 20334 consensus_queue.cc:984] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:22.851816 20334 consensus_queue.cc:984] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "252cea633d73492e8b9fb46364ebe78f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:22.894887 20334 consensus_queue.cc:984] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3810d972c23b409788635c10f7594962" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:23.152261 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:23.217424 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:23.217841 19320 tablet_replica.cc:273] T 052ace92eef0481cae05a81aec482e05 P 3810d972c23b409788635c10f7594962: stopping tablet replica I0918 20:38:23.218380 19320 raft_consensus.cc:2147] T 052ace92eef0481cae05a81aec482e05 P 3810d972c23b409788635c10f7594962 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:23.218667 19320 raft_consensus.cc:2174] T 052ace92eef0481cae05a81aec482e05 P 3810d972c23b409788635c10f7594962 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:23.222070 19320 tablet_replica.cc:273] T 319bc51fcd494e1b92a4837dcd658383 P 3810d972c23b409788635c10f7594962: stopping tablet replica I0918 20:38:23.222579 19320 raft_consensus.cc:2147] T 319bc51fcd494e1b92a4837dcd658383 P 3810d972c23b409788635c10f7594962 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:23.222848 19320 raft_consensus.cc:2174] T 319bc51fcd494e1b92a4837dcd658383 P 3810d972c23b409788635c10f7594962 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:23.225773 19320 tablet_replica.cc:273] T e46166b371974ac8b3f2bb853951c7a6 P 3810d972c23b409788635c10f7594962: stopping tablet replica I0918 20:38:23.226263 19320 raft_consensus.cc:2147] T e46166b371974ac8b3f2bb853951c7a6 P 3810d972c23b409788635c10f7594962 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:23.226553 19320 raft_consensus.cc:2174] T e46166b371974ac8b3f2bb853951c7a6 P 3810d972c23b409788635c10f7594962 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:23.243476 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:23.244444 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:23.244822 19320 tablet_server.cc:169] TabletServer shutdown complete. W0918 20:38:23.254926 20216 consensus_peers.cc:458] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 -> Peer 3810d972c23b409788635c10f7594962 (127.18.222.1:39537): Couldn't send request to peer 3810d972c23b409788635c10f7594962. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.1:39537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:38:23.256871 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:23.299278 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:23.299592 19320 tablet_replica.cc:273] T 052ace92eef0481cae05a81aec482e05 P 252cea633d73492e8b9fb46364ebe78f: stopping tablet replica I0918 20:38:23.299976 19320 raft_consensus.cc:2147] T 052ace92eef0481cae05a81aec482e05 P 252cea633d73492e8b9fb46364ebe78f [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:23.300268 19320 raft_consensus.cc:2174] T 052ace92eef0481cae05a81aec482e05 P 252cea633d73492e8b9fb46364ebe78f [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:23.303436 19320 tablet_replica.cc:273] T e46166b371974ac8b3f2bb853951c7a6 P 252cea633d73492e8b9fb46364ebe78f: stopping tablet replica I0918 20:38:23.304033 19320 raft_consensus.cc:2147] T e46166b371974ac8b3f2bb853951c7a6 P 252cea633d73492e8b9fb46364ebe78f [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:23.304402 19320 raft_consensus.cc:2174] T e46166b371974ac8b3f2bb853951c7a6 P 252cea633d73492e8b9fb46364ebe78f [term 1 FOLLOWER]: Raft consensus is shut down! W0918 20:38:23.305553 20216 consensus_peers.cc:458] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 -> Peer 3810d972c23b409788635c10f7594962 (127.18.222.1:39537): Couldn't send request to peer 3810d972c23b409788635c10f7594962. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.1:39537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:38:23.307467 19320 tablet_replica.cc:273] T 319bc51fcd494e1b92a4837dcd658383 P 252cea633d73492e8b9fb46364ebe78f: stopping tablet replica I0918 20:38:23.308007 19320 raft_consensus.cc:2147] T 319bc51fcd494e1b92a4837dcd658383 P 252cea633d73492e8b9fb46364ebe78f [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:23.308382 19320 raft_consensus.cc:2174] T 319bc51fcd494e1b92a4837dcd658383 P 252cea633d73492e8b9fb46364ebe78f [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:23.330569 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:23.330837 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:23.331266 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:23.341343 19320 tablet_server.cc:155] TabletServer shutting down... W0918 20:38:23.351250 20216 consensus_peers.cc:458] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 -> Peer 252cea633d73492e8b9fb46364ebe78f (127.18.222.2:42709): Couldn't send request to peer 252cea633d73492e8b9fb46364ebe78f. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.2:42709: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:38:23.373953 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:23.374277 19320 tablet_replica.cc:273] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4: stopping tablet replica I0918 20:38:23.374774 19320 raft_consensus.cc:2147] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:23.375424 19320 raft_consensus.cc:2174] T e46166b371974ac8b3f2bb853951c7a6 P 60645d4f7df445de88c17b9c461ebcc4 [term 1 FOLLOWER]: Raft consensus is shut down! W0918 20:38:23.376169 20216 consensus_peers.cc:458] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 -> Peer 3810d972c23b409788635c10f7594962 (127.18.222.1:39537): Couldn't send request to peer 3810d972c23b409788635c10f7594962. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.1:39537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:38:23.378383 19320 tablet_replica.cc:273] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4: stopping tablet replica I0918 20:38:23.378748 19320 raft_consensus.cc:2147] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:23.379390 19320 raft_consensus.cc:2174] T 319bc51fcd494e1b92a4837dcd658383 P 60645d4f7df445de88c17b9c461ebcc4 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:23.381708 19320 tablet_replica.cc:273] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4: stopping tablet replica I0918 20:38:23.382056 19320 raft_consensus.cc:2147] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:23.382622 19320 raft_consensus.cc:2174] T 052ace92eef0481cae05a81aec482e05 P 60645d4f7df445de88c17b9c461ebcc4 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:23.408427 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:23.408623 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:23.408957 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:23.417855 19320 master.cc:247] Master@127.18.222.62:33587 shutting down... I0918 20:38:23.433729 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:23.434088 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:23.434316 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 5fa8ea8cc088448b988deacd9000a0ad: stopping tablet replica I0918 20:38:23.452179 19320 master.cc:258] Master@127.18.222.62:33587 shutdown complete. I0918 20:38:23.459683 19320 master.cc:247] Master@127.18.222.61:45425 shutting down... I0918 20:38:23.477509 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:23.477852 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:23.478049 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 35ee39e0fecc41c0bb51dd5ccac4848b: stopping tablet replica I0918 20:38:23.495218 19320 master.cc:258] Master@127.18.222.61:45425 shutdown complete. I0918 20:38:23.502478 19320 master.cc:247] Master@127.18.222.60:44401 shutting down... I0918 20:38:23.519457 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:23.520175 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:23.520417 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 252ccf1ea83e476392b25d8272d26a03: stopping tablet replica I0918 20:38:23.537501 19320 master.cc:258] Master@127.18.222.60:44401 shutdown complete. [ OK ] RemoteKsckTest.TestCheckUnusualFlags (3013 ms) [ RUN ] RemoteKsckTest.TestTabletServerMismatchedUUID I0918 20:38:23.567338 19320 test_util.cc:212] Using random seed: -780511793 I0918 20:38:23.568092 19320 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.18.222.62:41123,127.18.222.61:44665,127.18.222.60:42463 I0918 20:38:23.580233 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 129092us I0918 20:38:23.580564 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:23.580703 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:23.580890 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:23.580999 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:23.581935 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/master-0-root/instance: uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" format_stamp: "Formatted at 2019-09-18 20:38:23 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:23.590288 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s I0918 20:38:23.591675 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:23.598103 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:23.598280 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/master-0-root uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" format_stamp: "Formatted at 2019-09-18 20:38:23 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:23.598505 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:23.652846 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:41123 I0918 20:38:23.654564 19320 webserver.cc:374] Webserver started at http://127.18.222.62:40545/ using document root and password file I0918 20:38:23.656419 20441 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:23.660847 20441 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:23.669674 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 129092us I0918 20:38:23.670033 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:23.670198 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:23.670507 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:23.670691 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:23.672080 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/master-1-root/instance: uuid: "6e70059559ab43c7ad0ec1fc928d667e" format_stamp: "Formatted at 2019-09-18 20:38:23 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:23.677737 20441 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } has no permanent_uuid. Determining permanent_uuid... W0918 20:38:23.679463 20441 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:44665: Network error: Client connection negotiation failed: client connection to 127.18.222.61:44665: connect: Connection refused (error 111) I0918 20:38:23.684010 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.010s sys 0.001s I0918 20:38:23.685559 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:23.693465 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:23.693663 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/master-1-root uuid: "6e70059559ab43c7ad0ec1fc928d667e" format_stamp: "Formatted at 2019-09-18 20:38:23 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:23.693940 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:23.740865 20441 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } attempt: 1 W0918 20:38:23.743991 20441 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:44665: Network error: Client connection negotiation failed: client connection to 127.18.222.61:44665: connect: Connection refused (error 111) I0918 20:38:23.759057 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:44665 I0918 20:38:23.760874 19320 webserver.cc:374] Webserver started at http://127.18.222.61:42739/ using document root and password file I0918 20:38:23.762557 20498 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:23.766863 20498 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:23.779254 20498 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:23.789607 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 129092us I0918 20:38:23.790113 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:23.790340 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:23.790632 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:23.790819 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:23.792121 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/master-2-root/instance: uuid: "893da579794d43378162b1bf6704dc54" format_stamp: "Formatted at 2019-09-18 20:38:23 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:23.794481 20498 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } has no permanent_uuid. Determining permanent_uuid... W0918 20:38:23.796267 20498 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:42463: Network error: Client connection negotiation failed: client connection to 127.18.222.60:42463: connect: Connection refused (error 111) I0918 20:38:23.798491 20441 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } attempt: 2 I0918 20:38:23.803426 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.011s sys 0.000s I0918 20:38:23.804996 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:23.808499 20441 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } has no permanent_uuid. Determining permanent_uuid... W0918 20:38:23.810089 20441 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:42463: Network error: Client connection negotiation failed: client connection to 127.18.222.60:42463: connect: Connection refused (error 111) I0918 20:38:23.813971 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:23.814190 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/master-2-root uuid: "893da579794d43378162b1bf6704dc54" format_stamp: "Formatted at 2019-09-18 20:38:23 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:23.814440 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:23.820637 20498 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } attempt: 1 W0918 20:38:23.822433 20498 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:42463: Network error: Client connection negotiation failed: client connection to 127.18.222.60:42463: connect: Connection refused (error 111) I0918 20:38:23.871553 20441 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } attempt: 1 W0918 20:38:23.873888 20441 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:42463: Network error: Client connection negotiation failed: client connection to 127.18.222.60:42463: connect: Connection refused (error 111) I0918 20:38:23.877355 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:42463 I0918 20:38:23.879084 19320 webserver.cc:374] Webserver started at http://127.18.222.60:45443/ using document root and password file I0918 20:38:23.880784 20555 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:23.883802 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0918 20:38:23.885257 20555 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:23.887787 20498 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } attempt: 2 I0918 20:38:23.898527 20555 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:23.906435 20498 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e: Bootstrap starting. I0918 20:38:23.909957 20498 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e: No blocks or log segments found. Creating new log. I0918 20:38:23.910111 20555 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:23.913292 20441 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } attempt: 2 I0918 20:38:23.914659 20498 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e: No bootstrap required, opened a new log I0918 20:38:23.921813 20498 raft_consensus.cc:340] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } I0918 20:38:23.922464 20498 raft_consensus.cc:366] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:23.922675 20498 raft_consensus.cc:705] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e70059559ab43c7ad0ec1fc928d667e, State: Initialized, Role: FOLLOWER I0918 20:38:23.925169 20498 consensus_queue.cc:229] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } I0918 20:38:23.926069 20560 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:23.926398 20560 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } I0918 20:38:23.929633 20498 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } } I0918 20:38:23.930410 20498 sys_catalog.cc:343] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:23.933961 20498 sys_catalog.cc:428] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [sys.catalog]: configured and running, proceeding with master startup. I0918 20:38:23.931186 20561 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } } I0918 20:38:23.932654 20560 leader_election.cc:296] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fc5f353671b4c1d8241bd0dd8e762b5 (127.18.222.62:41123), 893da579794d43378162b1bf6704dc54 (127.18.222.60:42463) I0918 20:38:23.935621 20555 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54: Bootstrap starting. I0918 20:38:23.939314 20555 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54: No blocks or log segments found. Creating new log. I0918 20:38:23.934859 20561 sys_catalog.cc:343] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:23.939405 20441 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5: Bootstrap starting. I0918 20:38:23.943352 20441 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5: No blocks or log segments found. Creating new log. W0918 20:38:23.944695 20562 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:38:23.944998 20562 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:23.945094 20555 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54: No bootstrap required, opened a new log I0918 20:38:23.948529 20555 raft_consensus.cc:340] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } I0918 20:38:23.949239 20555 raft_consensus.cc:366] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:23.949446 20555 raft_consensus.cc:705] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893da579794d43378162b1bf6704dc54, State: Initialized, Role: FOLLOWER I0918 20:38:23.949990 20555 consensus_queue.cc:229] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } I0918 20:38:23.951853 20555 sys_catalog.cc:340] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } } I0918 20:38:23.952564 20555 sys_catalog.cc:343] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:23.952616 20441 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5: No bootstrap required, opened a new log I0918 20:38:23.953807 20555 sys_catalog.cc:428] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:38:23.953918 20532 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e70059559ab43c7ad0ec1fc928d667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893da579794d43378162b1bf6704dc54" is_pre_election: true I0918 20:38:23.954668 20532 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e70059559ab43c7ad0ec1fc928d667e in term 0. I0918 20:38:23.952143 20565 sys_catalog.cc:340] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } } I0918 20:38:23.955176 20565 sys_catalog.cc:343] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:23.954973 20441 raft_consensus.cc:340] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } I0918 20:38:23.955986 20449 leader_election.cc:310] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e70059559ab43c7ad0ec1fc928d667e, 893da579794d43378162b1bf6704dc54; no voters: I0918 20:38:23.956200 20441 raft_consensus.cc:366] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:23.956454 20441 raft_consensus.cc:705] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc5f353671b4c1d8241bd0dd8e762b5, State: Initialized, Role: FOLLOWER I0918 20:38:23.957054 20561 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:23.957401 20561 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:23.957702 20561 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:23.957197 20441 consensus_queue.cc:229] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } W0918 20:38:23.957865 20566 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:38:23.958185 20566 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:23.959169 20441 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } } I0918 20:38:23.959357 20567 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } } I0918 20:38:23.959951 20441 sys_catalog.cc:343] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:23.960183 20567 sys_catalog.cc:343] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:23.961135 20441 sys_catalog.cc:428] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:38:23.961189 20418 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e70059559ab43c7ad0ec1fc928d667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" is_pre_election: true I0918 20:38:23.961832 20418 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e70059559ab43c7ad0ec1fc928d667e in term 0. I0918 20:38:23.962517 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 I0918 20:38:23.962786 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 I0918 20:38:23.963654 20561 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } W0918 20:38:23.964746 20568 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:38:23.964972 20568 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:23.965265 20561 leader_election.cc:296] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [CANDIDATE]: Term 1 election: Requested vote from peers 4fc5f353671b4c1d8241bd0dd8e762b5 (127.18.222.62:41123), 893da579794d43378162b1bf6704dc54 (127.18.222.60:42463) I0918 20:38:23.966079 20418 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e70059559ab43c7ad0ec1fc928d667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" I0918 20:38:23.966220 20532 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e70059559ab43c7ad0ec1fc928d667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893da579794d43378162b1bf6704dc54" I0918 20:38:23.966516 20418 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:23.966747 20532 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:23.970343 20418 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e70059559ab43c7ad0ec1fc928d667e in term 1. I0918 20:38:23.971348 20449 leader_election.cc:310] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc5f353671b4c1d8241bd0dd8e762b5, 6e70059559ab43c7ad0ec1fc928d667e; no voters: I0918 20:38:23.971840 20532 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e70059559ab43c7ad0ec1fc928d667e in term 1. I0918 20:38:23.972069 20561 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:23.972471 20561 raft_consensus.cc:667] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [term 1 LEADER]: Becoming Leader. State: Replica: 6e70059559ab43c7ad0ec1fc928d667e, State: Running, Role: LEADER I0918 20:38:23.973094 20561 consensus_queue.cc:206] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } I0918 20:38:23.976200 20560 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e70059559ab43c7ad0ec1fc928d667e. Latest consensus state: current_term: 1 leader_uuid: "6e70059559ab43c7ad0ec1fc928d667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } } I0918 20:38:23.977049 20560 sys_catalog.cc:343] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [sys.catalog]: This master's current role is: LEADER I0918 20:38:23.978498 20570 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:38:23.981094 20570 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:38:23.983863 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 129092us I0918 20:38:23.984228 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:23.984395 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:23.984608 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:23.984747 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:23.985723 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/ts-0-root/instance: uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" format_stamp: "Formatted at 2019-09-18 20:38:23 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:23.992177 20532 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54 [term 1 FOLLOWER]: Refusing update from remote peer 6e70059559ab43c7ad0ec1fc928d667e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:23.992236 20418 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5 [term 1 FOLLOWER]: Refusing update from remote peer 6e70059559ab43c7ad0ec1fc928d667e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:23.993532 20561 consensus_queue.cc:984] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:23.994174 20560 consensus_queue.cc:984] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:23.997670 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.012s user 0.013s sys 0.000s I0918 20:38:24.000289 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:24.006438 20567 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e70059559ab43c7ad0ec1fc928d667e. Latest consensus state: current_term: 1 leader_uuid: "6e70059559ab43c7ad0ec1fc928d667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } } I0918 20:38:24.007321 20567 sys_catalog.cc:343] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:24.009330 20565 sys_catalog.cc:340] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e70059559ab43c7ad0ec1fc928d667e. Latest consensus state: current_term: 1 leader_uuid: "6e70059559ab43c7ad0ec1fc928d667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } } I0918 20:38:24.010123 20565 sys_catalog.cc:343] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:24.016285 20561 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e70059559ab43c7ad0ec1fc928d667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } } I0918 20:38:24.017252 20561 sys_catalog.cc:343] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [sys.catalog]: This master's current role is: LEADER I0918 20:38:24.018072 20560 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e70059559ab43c7ad0ec1fc928d667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } } I0918 20:38:24.018913 20560 sys_catalog.cc:343] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [sys.catalog]: This master's current role is: LEADER I0918 20:38:24.021056 20565 sys_catalog.cc:340] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e70059559ab43c7ad0ec1fc928d667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } } I0918 20:38:24.022198 20565 sys_catalog.cc:343] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:24.023489 19320 fs_manager.cc:419] Time spent opening block manager: real 0.012s user 0.001s sys 0.000s I0918 20:38:24.023840 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/ts-0-root uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" format_stamp: "Formatted at 2019-09-18 20:38:23 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:24.023569 20567 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e70059559ab43c7ad0ec1fc928d667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc5f353671b4c1d8241bd0dd8e762b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 41123 } } peers { permanent_uuid: "6e70059559ab43c7ad0ec1fc928d667e" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44665 } } peers { permanent_uuid: "893da579794d43378162b1bf6704dc54" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42463 } } } I0918 20:38:24.024400 20567 sys_catalog.cc:343] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:24.024166 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:24.024533 20570 catalog_manager.cc:981] Generated new certificate authority record I0918 20:38:24.026837 20570 catalog_manager.cc:1106] Loading token signing keys... I0918 20:38:24.043534 20570 catalog_manager.cc:4446] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e: Generated new TSK 0 I0918 20:38:24.044278 20570 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:38:24.060832 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:24.061031 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:24.098315 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:45751 I0918 20:38:24.100157 19320 webserver.cc:374] Webserver started at http://127.18.222.1:36823/ using document root and password file I0918 20:38:24.122874 20644 heartbeater.cc:322] Connected to a master server at 127.18.222.60:42463 I0918 20:38:24.123261 20644 heartbeater.cc:406] Registering TS with master... I0918 20:38:24.123657 20645 heartbeater.cc:322] Connected to a master server at 127.18.222.61:44665 I0918 20:38:24.123952 20644 heartbeater.cc:455] Master 127.18.222.60:42463 requested a full tablet report, sending... I0918 20:38:24.124027 20645 heartbeater.cc:406] Registering TS with master... I0918 20:38:24.124408 20646 heartbeater.cc:322] Connected to a master server at 127.18.222.62:41123 I0918 20:38:24.124790 20646 heartbeater.cc:406] Registering TS with master... I0918 20:38:24.124791 20645 heartbeater.cc:455] Master 127.18.222.61:44665 requested a full tablet report, sending... I0918 20:38:24.125509 20646 heartbeater.cc:455] Master 127.18.222.62:41123 requested a full tablet report, sending... I0918 20:38:24.126900 20522 ts_manager.cc:186] Registered new tserver with Master: 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1:45751) I0918 20:38:24.127286 20465 ts_manager.cc:186] Registered new tserver with Master: 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1:45751) I0918 20:38:24.129566 20408 ts_manager.cc:186] Registered new tserver with Master: 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1:45751) I0918 20:38:24.130105 20465 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37576 I0918 20:38:24.131932 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 129592us I0918 20:38:24.132284 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:24.132444 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:24.132628 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:24.132741 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:24.133646 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/ts-1-root/instance: uuid: "413954693a164d5a80637ee06a7bf0b4" format_stamp: "Formatted at 2019-09-18 20:38:24 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:24.142900 20645 heartbeater.cc:447] Master 127.18.222.61:44665 was elected leader, sending a full tablet report... I0918 20:38:24.145185 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.010s sys 0.003s I0918 20:38:24.146844 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:24.155508 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s I0918 20:38:24.155773 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/ts-1-root uuid: "413954693a164d5a80637ee06a7bf0b4" format_stamp: "Formatted at 2019-09-18 20:38:24 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:24.156065 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:24.211086 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:24.211294 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:24.255894 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.2:37037 I0918 20:38:24.257776 19320 webserver.cc:374] Webserver started at http://127.18.222.2:46555/ using document root and password file I0918 20:38:24.290031 20718 heartbeater.cc:322] Connected to a master server at 127.18.222.62:41123 I0918 20:38:24.290356 20718 heartbeater.cc:406] Registering TS with master... I0918 20:38:24.290892 20718 heartbeater.cc:455] Master 127.18.222.62:41123 requested a full tablet report, sending... I0918 20:38:24.292975 20408 ts_manager.cc:186] Registered new tserver with Master: 413954693a164d5a80637ee06a7bf0b4 (127.18.222.2:37037) I0918 20:38:24.295492 20715 heartbeater.cc:322] Connected to a master server at 127.18.222.60:42463 I0918 20:38:24.295962 20715 heartbeater.cc:406] Registering TS with master... I0918 20:38:24.296550 20715 heartbeater.cc:455] Master 127.18.222.60:42463 requested a full tablet report, sending... I0918 20:38:24.297204 20716 heartbeater.cc:322] Connected to a master server at 127.18.222.61:44665 I0918 20:38:24.297524 20716 heartbeater.cc:406] Registering TS with master... I0918 20:38:24.298025 20716 heartbeater.cc:455] Master 127.18.222.61:44665 requested a full tablet report, sending... I0918 20:38:24.298933 20522 ts_manager.cc:186] Registered new tserver with Master: 413954693a164d5a80637ee06a7bf0b4 (127.18.222.2:37037) I0918 20:38:24.299670 20465 ts_manager.cc:186] Registered new tserver with Master: 413954693a164d5a80637ee06a7bf0b4 (127.18.222.2:37037) I0918 20:38:24.302304 20465 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37582 I0918 20:38:24.305390 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 129592us I0918 20:38:24.305922 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:24.306183 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:24.306465 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:24.306586 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:24.307938 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/ts-2-root/instance: uuid: "f72474f68c3248daad0f6975dd6bea9e" format_stamp: "Formatted at 2019-09-18 20:38:24 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:24.317073 20716 heartbeater.cc:447] Master 127.18.222.61:44665 was elected leader, sending a full tablet report... I0918 20:38:24.321638 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.013s user 0.013s sys 0.000s I0918 20:38:24.323204 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:24.335081 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s I0918 20:38:24.335309 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/ts-2-root uuid: "f72474f68c3248daad0f6975dd6bea9e" format_stamp: "Formatted at 2019-09-18 20:38:24 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:24.335557 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:24.387209 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:24.387490 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:24.454386 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.3:41199 I0918 20:38:24.456382 19320 webserver.cc:374] Webserver started at http://127.18.222.3:42413/ using document root and password file I0918 20:38:24.491219 20786 heartbeater.cc:322] Connected to a master server at 127.18.222.60:42463 I0918 20:38:24.491559 20786 heartbeater.cc:406] Registering TS with master... I0918 20:38:24.495808 20786 heartbeater.cc:455] Master 127.18.222.60:42463 requested a full tablet report, sending... I0918 20:38:24.498311 20522 ts_manager.cc:186] Registered new tserver with Master: f72474f68c3248daad0f6975dd6bea9e (127.18.222.3:41199) I0918 20:38:24.607491 20787 heartbeater.cc:322] Connected to a master server at 127.18.222.61:44665 I0918 20:38:24.607918 20787 heartbeater.cc:406] Registering TS with master... I0918 20:38:24.608475 20787 heartbeater.cc:455] Master 127.18.222.61:44665 requested a full tablet report, sending... I0918 20:38:24.610404 20465 ts_manager.cc:186] Registered new tserver with Master: f72474f68c3248daad0f6975dd6bea9e (127.18.222.3:41199) I0918 20:38:24.612354 20465 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37588 I0918 20:38:24.614363 20789 heartbeater.cc:322] Connected to a master server at 127.18.222.62:41123 I0918 20:38:24.614694 20789 heartbeater.cc:406] Registering TS with master... I0918 20:38:24.615240 20789 heartbeater.cc:455] Master 127.18.222.62:41123 requested a full tablet report, sending... I0918 20:38:24.617760 20408 ts_manager.cc:186] Registered new tserver with Master: f72474f68c3248daad0f6975dd6bea9e (127.18.222.3:41199) I0918 20:38:24.618999 19320 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.120899094s I0918 20:38:24.625177 20787 heartbeater.cc:447] Master 127.18.222.61:44665 was elected leader, sending a full tablet report... I0918 20:38:24.671216 20465 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37592: name: "ksck-test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\n\000\000\000\004\001\024\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:38:24.673493 20465 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:38:24.815584 20682 tablet_service.cc:1022] Processing CreateTablet for tablet 51e4b66c13a1478b813c2f9b8af9f1e9 (table=ksck-test-table [id=df736ecc0a6f4a13bac1334e98f07d1c]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:24.819296 20682 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 51e4b66c13a1478b813c2f9b8af9f1e9. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:24.831909 20753 tablet_service.cc:1022] Processing CreateTablet for tablet 51e4b66c13a1478b813c2f9b8af9f1e9 (table=ksck-test-table [id=df736ecc0a6f4a13bac1334e98f07d1c]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:24.832973 20680 tablet_service.cc:1022] Processing CreateTablet for tablet 199cb07c60bf471a801872c84de7b218 (table=ksck-test-table [id=df736ecc0a6f4a13bac1334e98f07d1c]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:24.831909 20681 tablet_service.cc:1022] Processing CreateTablet for tablet f7ed278cf9394be89b6ddd9b49a861bb (table=ksck-test-table [id=df736ecc0a6f4a13bac1334e98f07d1c]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:24.833971 20680 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 199cb07c60bf471a801872c84de7b218. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:24.833035 20753 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 51e4b66c13a1478b813c2f9b8af9f1e9. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:24.836737 20681 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f7ed278cf9394be89b6ddd9b49a861bb. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:24.837188 20752 tablet_service.cc:1022] Processing CreateTablet for tablet f7ed278cf9394be89b6ddd9b49a861bb (table=ksck-test-table [id=df736ecc0a6f4a13bac1334e98f07d1c]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:24.838145 20752 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f7ed278cf9394be89b6ddd9b49a861bb. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:24.845661 20751 tablet_service.cc:1022] Processing CreateTablet for tablet 199cb07c60bf471a801872c84de7b218 (table=ksck-test-table [id=df736ecc0a6f4a13bac1334e98f07d1c]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:24.846742 20751 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 199cb07c60bf471a801872c84de7b218. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:24.848825 20611 tablet_service.cc:1022] Processing CreateTablet for tablet 51e4b66c13a1478b813c2f9b8af9f1e9 (table=ksck-test-table [id=df736ecc0a6f4a13bac1334e98f07d1c]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:24.849514 20610 tablet_service.cc:1022] Processing CreateTablet for tablet f7ed278cf9394be89b6ddd9b49a861bb (table=ksck-test-table [id=df736ecc0a6f4a13bac1334e98f07d1c]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:24.849844 20611 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 51e4b66c13a1478b813c2f9b8af9f1e9. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:24.850505 20610 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f7ed278cf9394be89b6ddd9b49a861bb. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:24.852397 20609 tablet_service.cc:1022] Processing CreateTablet for tablet 199cb07c60bf471a801872c84de7b218 (table=ksck-test-table [id=df736ecc0a6f4a13bac1334e98f07d1c]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:24.853340 20609 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 199cb07c60bf471a801872c84de7b218. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:24.890487 20813 tablet_bootstrap.cc:439] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e: Bootstrap starting. I0918 20:38:24.890974 20815 tablet_bootstrap.cc:439] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9: Bootstrap starting. I0918 20:38:24.892772 20814 tablet_bootstrap.cc:439] T 199cb07c60bf471a801872c84de7b218 P 413954693a164d5a80637ee06a7bf0b4: Bootstrap starting. I0918 20:38:24.894582 20813 tablet_bootstrap.cc:590] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e: No blocks or log segments found. Creating new log. I0918 20:38:24.898106 20815 tablet_bootstrap.cc:590] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9: No blocks or log segments found. Creating new log. I0918 20:38:24.905699 20814 tablet_bootstrap.cc:590] T 199cb07c60bf471a801872c84de7b218 P 413954693a164d5a80637ee06a7bf0b4: No blocks or log segments found. Creating new log. I0918 20:38:24.979418 20815 tablet_bootstrap.cc:439] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9: No bootstrap required, opened a new log I0918 20:38:24.980034 20815 ts_tablet_manager.cc:1085] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9: Time spent bootstrapping tablet: real 0.089s user 0.035s sys 0.042s I0918 20:38:24.981443 20566 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54: acquiring CA information for follower catalog manager: success I0918 20:38:24.982079 20815 raft_consensus.cc:340] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } I0918 20:38:24.982815 20815 raft_consensus.cc:366] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:24.983045 20815 raft_consensus.cc:705] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 644cb35e677e49c3a2210fd6b3f4f8b9, State: Initialized, Role: FOLLOWER I0918 20:38:24.983583 20815 consensus_queue.cc:229] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } I0918 20:38:24.985455 20813 tablet_bootstrap.cc:439] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e: No bootstrap required, opened a new log I0918 20:38:24.988072 20813 ts_tablet_manager.cc:1085] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e: Time spent bootstrapping tablet: real 0.098s user 0.007s sys 0.003s I0918 20:38:24.990082 20813 raft_consensus.cc:340] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } I0918 20:38:24.990782 20813 raft_consensus.cc:366] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:24.991024 20813 raft_consensus.cc:705] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f72474f68c3248daad0f6975dd6bea9e, State: Initialized, Role: FOLLOWER I0918 20:38:24.991869 20568 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5: acquiring CA information for follower catalog manager: success I0918 20:38:24.992924 20814 tablet_bootstrap.cc:439] T 199cb07c60bf471a801872c84de7b218 P 413954693a164d5a80637ee06a7bf0b4: No bootstrap required, opened a new log I0918 20:38:24.993403 20814 ts_tablet_manager.cc:1085] T 199cb07c60bf471a801872c84de7b218 P 413954693a164d5a80637ee06a7bf0b4: Time spent bootstrapping tablet: real 0.101s user 0.035s sys 0.040s I0918 20:38:24.991600 20813 consensus_queue.cc:229] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } I0918 20:38:24.995998 20814 raft_consensus.cc:340] T 199cb07c60bf471a801872c84de7b218 P 413954693a164d5a80637ee06a7bf0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } I0918 20:38:24.997226 20814 raft_consensus.cc:366] T 199cb07c60bf471a801872c84de7b218 P 413954693a164d5a80637ee06a7bf0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:24.997550 20814 raft_consensus.cc:705] T 199cb07c60bf471a801872c84de7b218 P 413954693a164d5a80637ee06a7bf0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413954693a164d5a80637ee06a7bf0b4, State: Initialized, Role: FOLLOWER I0918 20:38:25.004653 20568 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:38:25.006639 20813 ts_tablet_manager.cc:1113] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s I0918 20:38:25.007485 20813 tablet_bootstrap.cc:439] T 199cb07c60bf471a801872c84de7b218 P f72474f68c3248daad0f6975dd6bea9e: Bootstrap starting. I0918 20:38:25.010697 20813 tablet_bootstrap.cc:590] T 199cb07c60bf471a801872c84de7b218 P f72474f68c3248daad0f6975dd6bea9e: No blocks or log segments found. Creating new log. I0918 20:38:25.015645 20814 consensus_queue.cc:229] T 199cb07c60bf471a801872c84de7b218 P 413954693a164d5a80637ee06a7bf0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } I0918 20:38:25.023461 20566 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:38:25.015686 20815 ts_tablet_manager.cc:1113] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9: Time spent starting tablet: real 0.035s user 0.017s sys 0.011s I0918 20:38:25.024663 20815 tablet_bootstrap.cc:439] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9: Bootstrap starting. I0918 20:38:25.028239 20815 tablet_bootstrap.cc:590] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9: No blocks or log segments found. Creating new log. I0918 20:38:25.035579 20815 tablet_bootstrap.cc:439] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9: No bootstrap required, opened a new log I0918 20:38:25.036329 20815 ts_tablet_manager.cc:1085] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s I0918 20:38:25.038677 20815 raft_consensus.cc:340] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } I0918 20:38:25.039500 20815 raft_consensus.cc:366] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:25.039804 20815 raft_consensus.cc:705] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 644cb35e677e49c3a2210fd6b3f4f8b9, State: Initialized, Role: FOLLOWER I0918 20:38:25.040649 20815 consensus_queue.cc:229] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } I0918 20:38:25.042523 20813 tablet_bootstrap.cc:439] T 199cb07c60bf471a801872c84de7b218 P f72474f68c3248daad0f6975dd6bea9e: No bootstrap required, opened a new log I0918 20:38:25.043078 20813 ts_tablet_manager.cc:1085] T 199cb07c60bf471a801872c84de7b218 P f72474f68c3248daad0f6975dd6bea9e: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.000s I0918 20:38:25.046608 20819 raft_consensus.cc:472] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:25.047385 20813 raft_consensus.cc:340] T 199cb07c60bf471a801872c84de7b218 P f72474f68c3248daad0f6975dd6bea9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } I0918 20:38:25.048271 20813 raft_consensus.cc:366] T 199cb07c60bf471a801872c84de7b218 P f72474f68c3248daad0f6975dd6bea9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:25.048532 20813 raft_consensus.cc:705] T 199cb07c60bf471a801872c84de7b218 P f72474f68c3248daad0f6975dd6bea9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f72474f68c3248daad0f6975dd6bea9e, State: Initialized, Role: FOLLOWER I0918 20:38:25.047381 20814 ts_tablet_manager.cc:1113] T 199cb07c60bf471a801872c84de7b218 P 413954693a164d5a80637ee06a7bf0b4: Time spent starting tablet: real 0.054s user 0.006s sys 0.010s I0918 20:38:25.049263 20813 consensus_queue.cc:229] T 199cb07c60bf471a801872c84de7b218 P f72474f68c3248daad0f6975dd6bea9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } I0918 20:38:25.050001 20814 tablet_bootstrap.cc:439] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 413954693a164d5a80637ee06a7bf0b4: Bootstrap starting. I0918 20:38:25.047083 20819 raft_consensus.cc:494] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } I0918 20:38:25.054039 20814 tablet_bootstrap.cc:590] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 413954693a164d5a80637ee06a7bf0b4: No blocks or log segments found. Creating new log. I0918 20:38:25.055579 20815 ts_tablet_manager.cc:1113] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9: Time spent starting tablet: real 0.019s user 0.003s sys 0.001s I0918 20:38:25.056766 20815 tablet_bootstrap.cc:439] T f7ed278cf9394be89b6ddd9b49a861bb P 644cb35e677e49c3a2210fd6b3f4f8b9: Bootstrap starting. I0918 20:38:25.068009 20815 tablet_bootstrap.cc:590] T f7ed278cf9394be89b6ddd9b49a861bb P 644cb35e677e49c3a2210fd6b3f4f8b9: No blocks or log segments found. Creating new log. I0918 20:38:25.069005 20819 leader_election.cc:296] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f72474f68c3248daad0f6975dd6bea9e (127.18.222.3:41199), 413954693a164d5a80637ee06a7bf0b4 (127.18.222.2:37037) I0918 20:38:25.097800 20813 ts_tablet_manager.cc:1113] T 199cb07c60bf471a801872c84de7b218 P f72474f68c3248daad0f6975dd6bea9e: Time spent starting tablet: real 0.054s user 0.006s sys 0.000s I0918 20:38:25.098806 20813 tablet_bootstrap.cc:439] T 51e4b66c13a1478b813c2f9b8af9f1e9 P f72474f68c3248daad0f6975dd6bea9e: Bootstrap starting. I0918 20:38:25.102448 20814 tablet_bootstrap.cc:439] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 413954693a164d5a80637ee06a7bf0b4: No bootstrap required, opened a new log I0918 20:38:25.103035 20814 ts_tablet_manager.cc:1085] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 413954693a164d5a80637ee06a7bf0b4: Time spent bootstrapping tablet: real 0.053s user 0.019s sys 0.023s I0918 20:38:25.118513 20813 tablet_bootstrap.cc:590] T 51e4b66c13a1478b813c2f9b8af9f1e9 P f72474f68c3248daad0f6975dd6bea9e: No blocks or log segments found. Creating new log. I0918 20:38:25.118311 20814 raft_consensus.cc:340] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 413954693a164d5a80637ee06a7bf0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } I0918 20:38:25.119895 20814 raft_consensus.cc:366] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 413954693a164d5a80637ee06a7bf0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:25.120548 20814 raft_consensus.cc:705] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 413954693a164d5a80637ee06a7bf0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413954693a164d5a80637ee06a7bf0b4, State: Initialized, Role: FOLLOWER I0918 20:38:25.126067 20814 consensus_queue.cc:229] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 413954693a164d5a80637ee06a7bf0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } I0918 20:38:25.129719 20814 ts_tablet_manager.cc:1113] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 413954693a164d5a80637ee06a7bf0b4: Time spent starting tablet: real 0.026s user 0.006s sys 0.002s I0918 20:38:25.130550 20814 tablet_bootstrap.cc:439] T f7ed278cf9394be89b6ddd9b49a861bb P 413954693a164d5a80637ee06a7bf0b4: Bootstrap starting. I0918 20:38:25.133374 20692 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "51e4b66c13a1478b813c2f9b8af9f1e9" candidate_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413954693a164d5a80637ee06a7bf0b4" is_pre_election: true I0918 20:38:25.134091 20814 tablet_bootstrap.cc:590] T f7ed278cf9394be89b6ddd9b49a861bb P 413954693a164d5a80637ee06a7bf0b4: No blocks or log segments found. Creating new log. I0918 20:38:25.134176 20692 raft_consensus.cc:2350] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 413954693a164d5a80637ee06a7bf0b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 644cb35e677e49c3a2210fd6b3f4f8b9 in term 0. I0918 20:38:25.139685 20581 leader_election.cc:310] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413954693a164d5a80637ee06a7bf0b4, 644cb35e677e49c3a2210fd6b3f4f8b9; no voters: I0918 20:38:25.141389 20819 raft_consensus.cc:2644] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:25.141718 20819 raft_consensus.cc:472] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:25.142047 20819 raft_consensus.cc:2890] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:25.152709 20819 raft_consensus.cc:494] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } I0918 20:38:25.158655 20819 leader_election.cc:296] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [CANDIDATE]: Term 1 election: Requested vote from peers f72474f68c3248daad0f6975dd6bea9e (127.18.222.3:41199), 413954693a164d5a80637ee06a7bf0b4 (127.18.222.2:37037) I0918 20:38:25.159617 20692 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "51e4b66c13a1478b813c2f9b8af9f1e9" candidate_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413954693a164d5a80637ee06a7bf0b4" I0918 20:38:25.162915 20692 raft_consensus.cc:2890] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 413954693a164d5a80637ee06a7bf0b4 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:25.180383 20815 tablet_bootstrap.cc:439] T f7ed278cf9394be89b6ddd9b49a861bb P 644cb35e677e49c3a2210fd6b3f4f8b9: No bootstrap required, opened a new log I0918 20:38:25.183243 20815 ts_tablet_manager.cc:1085] T f7ed278cf9394be89b6ddd9b49a861bb P 644cb35e677e49c3a2210fd6b3f4f8b9: Time spent bootstrapping tablet: real 0.127s user 0.019s sys 0.003s I0918 20:38:25.186888 20814 tablet_bootstrap.cc:439] T f7ed278cf9394be89b6ddd9b49a861bb P 413954693a164d5a80637ee06a7bf0b4: No bootstrap required, opened a new log I0918 20:38:25.187918 20814 ts_tablet_manager.cc:1085] T f7ed278cf9394be89b6ddd9b49a861bb P 413954693a164d5a80637ee06a7bf0b4: Time spent bootstrapping tablet: real 0.058s user 0.013s sys 0.012s I0918 20:38:25.190510 20814 raft_consensus.cc:340] T f7ed278cf9394be89b6ddd9b49a861bb P 413954693a164d5a80637ee06a7bf0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } I0918 20:38:25.190913 20815 raft_consensus.cc:340] T f7ed278cf9394be89b6ddd9b49a861bb P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } I0918 20:38:25.192025 20815 raft_consensus.cc:366] T f7ed278cf9394be89b6ddd9b49a861bb P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:25.191257 20814 raft_consensus.cc:366] T f7ed278cf9394be89b6ddd9b49a861bb P 413954693a164d5a80637ee06a7bf0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:25.193835 20814 raft_consensus.cc:705] T f7ed278cf9394be89b6ddd9b49a861bb P 413954693a164d5a80637ee06a7bf0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413954693a164d5a80637ee06a7bf0b4, State: Initialized, Role: FOLLOWER I0918 20:38:25.197113 20819 raft_consensus.cc:472] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:25.197609 20819 raft_consensus.cc:494] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } I0918 20:38:25.200206 20691 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "199cb07c60bf471a801872c84de7b218" candidate_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413954693a164d5a80637ee06a7bf0b4" is_pre_election: true I0918 20:38:25.200871 20691 raft_consensus.cc:2350] T 199cb07c60bf471a801872c84de7b218 P 413954693a164d5a80637ee06a7bf0b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 644cb35e677e49c3a2210fd6b3f4f8b9 in term 0. I0918 20:38:25.202009 20581 leader_election.cc:310] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413954693a164d5a80637ee06a7bf0b4, 644cb35e677e49c3a2210fd6b3f4f8b9; no voters: I0918 20:38:25.193563 20813 tablet_bootstrap.cc:439] T 51e4b66c13a1478b813c2f9b8af9f1e9 P f72474f68c3248daad0f6975dd6bea9e: No bootstrap required, opened a new log I0918 20:38:25.203210 20813 ts_tablet_manager.cc:1085] T 51e4b66c13a1478b813c2f9b8af9f1e9 P f72474f68c3248daad0f6975dd6bea9e: Time spent bootstrapping tablet: real 0.105s user 0.028s sys 0.020s I0918 20:38:25.204582 20763 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "51e4b66c13a1478b813c2f9b8af9f1e9" candidate_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f72474f68c3248daad0f6975dd6bea9e" is_pre_election: true I0918 20:38:25.205806 20762 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "51e4b66c13a1478b813c2f9b8af9f1e9" candidate_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f72474f68c3248daad0f6975dd6bea9e" I0918 20:38:25.206037 20813 raft_consensus.cc:340] T 51e4b66c13a1478b813c2f9b8af9f1e9 P f72474f68c3248daad0f6975dd6bea9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } I0918 20:38:25.206773 20813 raft_consensus.cc:366] T 51e4b66c13a1478b813c2f9b8af9f1e9 P f72474f68c3248daad0f6975dd6bea9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely W0918 20:38:25.206955 20582 leader_election.cc:348] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f72474f68c3248daad0f6975dd6bea9e (127.18.222.3:41199): Illegal state: must be running to vote when last-logged opid is not known I0918 20:38:25.207801 20692 raft_consensus.cc:2350] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 413954693a164d5a80637ee06a7bf0b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 644cb35e677e49c3a2210fd6b3f4f8b9 in term 1. I0918 20:38:25.194550 20814 consensus_queue.cc:229] T f7ed278cf9394be89b6ddd9b49a861bb P 413954693a164d5a80637ee06a7bf0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } I0918 20:38:25.192281 20815 raft_consensus.cc:705] T f7ed278cf9394be89b6ddd9b49a861bb P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 644cb35e677e49c3a2210fd6b3f4f8b9, State: Initialized, Role: FOLLOWER I0918 20:38:25.210289 20763 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "199cb07c60bf471a801872c84de7b218" candidate_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f72474f68c3248daad0f6975dd6bea9e" is_pre_election: true I0918 20:38:25.211287 20815 consensus_queue.cc:229] T f7ed278cf9394be89b6ddd9b49a861bb P 644cb35e677e49c3a2210fd6b3f4f8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } I0918 20:38:25.207031 20813 raft_consensus.cc:705] T 51e4b66c13a1478b813c2f9b8af9f1e9 P f72474f68c3248daad0f6975dd6bea9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f72474f68c3248daad0f6975dd6bea9e, State: Initialized, Role: FOLLOWER I0918 20:38:25.210913 20763 raft_consensus.cc:2350] T 199cb07c60bf471a801872c84de7b218 P f72474f68c3248daad0f6975dd6bea9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 644cb35e677e49c3a2210fd6b3f4f8b9 in term 0. I0918 20:38:25.212954 20819 leader_election.cc:296] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 413954693a164d5a80637ee06a7bf0b4 (127.18.222.2:37037), f72474f68c3248daad0f6975dd6bea9e (127.18.222.3:41199) I0918 20:38:25.220296 20819 raft_consensus.cc:2644] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:25.218499 20813 consensus_queue.cc:229] T 51e4b66c13a1478b813c2f9b8af9f1e9 P f72474f68c3248daad0f6975dd6bea9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } I0918 20:38:25.215854 20581 leader_election.cc:310] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413954693a164d5a80637ee06a7bf0b4, 644cb35e677e49c3a2210fd6b3f4f8b9; no voters: I0918 20:38:25.225872 20819 raft_consensus.cc:472] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:25.229686 20819 raft_consensus.cc:2890] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:25.229048 20762 raft_consensus.cc:2890] T 51e4b66c13a1478b813c2f9b8af9f1e9 P f72474f68c3248daad0f6975dd6bea9e [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:25.231753 20815 ts_tablet_manager.cc:1113] T f7ed278cf9394be89b6ddd9b49a861bb P 644cb35e677e49c3a2210fd6b3f4f8b9: Time spent starting tablet: real 0.046s user 0.015s sys 0.006s I0918 20:38:25.232219 20830 raft_consensus.cc:2644] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:25.235937 20813 ts_tablet_manager.cc:1113] T 51e4b66c13a1478b813c2f9b8af9f1e9 P f72474f68c3248daad0f6975dd6bea9e: Time spent starting tablet: real 0.032s user 0.006s sys 0.001s I0918 20:38:25.244117 20819 raft_consensus.cc:494] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } I0918 20:38:25.246527 20692 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "199cb07c60bf471a801872c84de7b218" candidate_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413954693a164d5a80637ee06a7bf0b4" I0918 20:38:25.246858 20763 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "199cb07c60bf471a801872c84de7b218" candidate_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f72474f68c3248daad0f6975dd6bea9e" I0918 20:38:25.247485 20763 raft_consensus.cc:2890] T 199cb07c60bf471a801872c84de7b218 P f72474f68c3248daad0f6975dd6bea9e [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:25.248127 20762 raft_consensus.cc:2350] T 51e4b66c13a1478b813c2f9b8af9f1e9 P f72474f68c3248daad0f6975dd6bea9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 644cb35e677e49c3a2210fd6b3f4f8b9 in term 1. I0918 20:38:25.247109 20692 raft_consensus.cc:2890] T 199cb07c60bf471a801872c84de7b218 P 413954693a164d5a80637ee06a7bf0b4 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:25.258711 20692 raft_consensus.cc:2350] T 199cb07c60bf471a801872c84de7b218 P 413954693a164d5a80637ee06a7bf0b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 644cb35e677e49c3a2210fd6b3f4f8b9 in term 1. I0918 20:38:25.259933 20814 ts_tablet_manager.cc:1113] T f7ed278cf9394be89b6ddd9b49a861bb P 413954693a164d5a80637ee06a7bf0b4: Time spent starting tablet: real 0.072s user 0.006s sys 0.000s I0918 20:38:25.259605 20830 raft_consensus.cc:667] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 1 LEADER]: Becoming Leader. State: Replica: 644cb35e677e49c3a2210fd6b3f4f8b9, State: Running, Role: LEADER I0918 20:38:25.260963 20830 consensus_queue.cc:206] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } I0918 20:38:25.263352 20763 raft_consensus.cc:2350] T 199cb07c60bf471a801872c84de7b218 P f72474f68c3248daad0f6975dd6bea9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 644cb35e677e49c3a2210fd6b3f4f8b9 in term 1. I0918 20:38:25.264087 20819 leader_election.cc:296] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 [CANDIDATE]: Term 1 election: Requested vote from peers 413954693a164d5a80637ee06a7bf0b4 (127.18.222.2:37037), f72474f68c3248daad0f6975dd6bea9e (127.18.222.3:41199) I0918 20:38:25.261636 20581 leader_election.cc:310] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413954693a164d5a80637ee06a7bf0b4, 644cb35e677e49c3a2210fd6b3f4f8b9; no voters: I0918 20:38:25.267170 20819 raft_consensus.cc:2644] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:25.268913 20819 raft_consensus.cc:667] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 1 LEADER]: Becoming Leader. State: Replica: 644cb35e677e49c3a2210fd6b3f4f8b9, State: Running, Role: LEADER I0918 20:38:25.269603 20819 consensus_queue.cc:206] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } I0918 20:38:25.274910 20464 catalog_manager.cc:4131] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 reported cstate change: term changed from 0 to 1, leader changed from to 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1). New cstate: current_term: 1 leader_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } health_report { overall_health: UNKNOWN } } } I0918 20:38:25.345933 20820 raft_consensus.cc:472] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:25.346383 20820 raft_consensus.cc:494] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } I0918 20:38:25.349277 20820 leader_election.cc:296] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1:45751), 413954693a164d5a80637ee06a7bf0b4 (127.18.222.2:37037) I0918 20:38:25.434350 20463 catalog_manager.cc:4131] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 reported cstate change: term changed from 0 to 1, leader changed from to 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1). New cstate: current_term: 1 leader_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } health_report { overall_health: UNKNOWN } } } I0918 20:38:25.442247 20692 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f7ed278cf9394be89b6ddd9b49a861bb" candidate_uuid: "f72474f68c3248daad0f6975dd6bea9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413954693a164d5a80637ee06a7bf0b4" is_pre_election: true I0918 20:38:25.442943 20692 raft_consensus.cc:2350] T f7ed278cf9394be89b6ddd9b49a861bb P 413954693a164d5a80637ee06a7bf0b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f72474f68c3248daad0f6975dd6bea9e in term 0. I0918 20:38:25.443966 20724 leader_election.cc:310] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413954693a164d5a80637ee06a7bf0b4, f72474f68c3248daad0f6975dd6bea9e; no voters: I0918 20:38:25.444890 20820 raft_consensus.cc:2644] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:25.445214 20820 raft_consensus.cc:472] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:25.445509 20820 raft_consensus.cc:2890] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:25.451318 20820 raft_consensus.cc:494] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } I0918 20:38:25.454103 20820 leader_election.cc:296] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [CANDIDATE]: Term 1 election: Requested vote from peers 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1:45751), 413954693a164d5a80637ee06a7bf0b4 (127.18.222.2:37037) I0918 20:38:25.454303 20692 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f7ed278cf9394be89b6ddd9b49a861bb" candidate_uuid: "f72474f68c3248daad0f6975dd6bea9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413954693a164d5a80637ee06a7bf0b4" I0918 20:38:25.455147 20692 raft_consensus.cc:2890] T f7ed278cf9394be89b6ddd9b49a861bb P 413954693a164d5a80637ee06a7bf0b4 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:25.463948 20692 raft_consensus.cc:2350] T f7ed278cf9394be89b6ddd9b49a861bb P 413954693a164d5a80637ee06a7bf0b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f72474f68c3248daad0f6975dd6bea9e in term 1. I0918 20:38:25.465500 20724 leader_election.cc:310] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413954693a164d5a80637ee06a7bf0b4, f72474f68c3248daad0f6975dd6bea9e; no voters: I0918 20:38:25.466440 20820 raft_consensus.cc:2644] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:25.473603 20621 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f7ed278cf9394be89b6ddd9b49a861bb" candidate_uuid: "f72474f68c3248daad0f6975dd6bea9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" is_pre_election: true I0918 20:38:25.474385 20621 raft_consensus.cc:2350] T f7ed278cf9394be89b6ddd9b49a861bb P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f72474f68c3248daad0f6975dd6bea9e in term 0. I0918 20:38:25.475452 20620 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f7ed278cf9394be89b6ddd9b49a861bb" candidate_uuid: "f72474f68c3248daad0f6975dd6bea9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" I0918 20:38:25.476151 20620 raft_consensus.cc:2890] T f7ed278cf9394be89b6ddd9b49a861bb P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:25.484560 20620 raft_consensus.cc:2350] T f7ed278cf9394be89b6ddd9b49a861bb P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f72474f68c3248daad0f6975dd6bea9e in term 1. I0918 20:38:25.489528 20820 raft_consensus.cc:667] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [term 1 LEADER]: Becoming Leader. State: Replica: f72474f68c3248daad0f6975dd6bea9e, State: Running, Role: LEADER I0918 20:38:25.490095 20820 consensus_queue.cc:206] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } } peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } I0918 20:38:25.497090 20462 catalog_manager.cc:4131] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e reported cstate change: term changed from 0 to 1, leader changed from to f72474f68c3248daad0f6975dd6bea9e (127.18.222.3). New cstate: current_term: 1 leader_uuid: "f72474f68c3248daad0f6975dd6bea9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } health_report { overall_health: HEALTHY } } } I0918 20:38:25.708206 20819 consensus_queue.cc:984] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:25.765720 20819 consensus_queue.cc:984] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:25.772509 20819 consensus_queue.cc:984] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:25.790035 20819 consensus_queue.cc:984] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:25.893620 20820 consensus_queue.cc:984] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:25.916435 20845 consensus_queue.cc:984] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s I0918 20:38:26.080756 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:26.140720 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:26.141109 19320 tablet_replica.cc:273] T f7ed278cf9394be89b6ddd9b49a861bb P 644cb35e677e49c3a2210fd6b3f4f8b9: stopping tablet replica I0918 20:38:26.141610 19320 raft_consensus.cc:2147] T f7ed278cf9394be89b6ddd9b49a861bb P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:26.141993 19320 raft_consensus.cc:2174] T f7ed278cf9394be89b6ddd9b49a861bb P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:26.145634 19320 tablet_replica.cc:273] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9: stopping tablet replica I0918 20:38:26.146001 19320 raft_consensus.cc:2147] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:26.146755 19320 raft_consensus.cc:2174] T 199cb07c60bf471a801872c84de7b218 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:26.149255 19320 tablet_replica.cc:273] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9: stopping tablet replica I0918 20:38:26.149575 19320 raft_consensus.cc:2147] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:26.150135 19320 raft_consensus.cc:2174] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 644cb35e677e49c3a2210fd6b3f4f8b9 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:26.173002 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:26.173205 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:26.173544 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:26.204037 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 130592us I0918 20:38:26.204540 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:26.204699 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:26.204932 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:26.205058 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:26.206184 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/ts-0-root2/instance: uuid: "d7a56d8c2b174760bbaefaa16dc0ff8d" format_stamp: "Formatted at 2019-09-18 20:38:26 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:26.220394 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.014s user 0.016s sys 0.000s I0918 20:38:26.222770 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:26.233005 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:26.233220 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/ts-0-root2 uuid: "d7a56d8c2b174760bbaefaa16dc0ff8d" format_stamp: "Formatted at 2019-09-18 20:38:26 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:26.233484 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/ts-0-root2 metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/ts-0-root2 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1568839089958813-19320/minicluster-data/ts-0-root2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:26.272015 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:26.272244 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:26.326514 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:45751 I0918 20:38:26.329951 19320 webserver.cc:374] Webserver started at http://127.18.222.1:41787/ using document root and password file W0918 20:38:26.367055 20953 heartbeater.cc:571] Failed to heartbeat to 127.0.0.1:7051 (0 consecutive failures): Network error: Failed to ping master at 127.0.0.1:7051: Client connection negotiation failed: client connection to 127.0.0.1:7051: connect: Connection refused (error 111) W0918 20:38:26.392169 20953 heartbeater.cc:357] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. W0918 20:38:26.485008 20930 tablet_service.cc:330] Invalid argument: UpdateConsensus: Wrong destination UUID requested. Local UUID: d7a56d8c2b174760bbaefaa16dc0ff8d. Requested UUID: 644cb35e677e49c3a2210fd6b3f4f8b9: from {username='slave'} at 127.0.0.1:52718: tablet_id: "f7ed278cf9394be89b6ddd9b49a861bb" caller_uuid: "f72474f68c3248daad0f6975dd6bea9e" caller_term: 1 preceding_id { term: 1 index: 1 } dest_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" committed_index: 1 all_replicated_index: 1 safe_timestamp: 6425964980100636672 last_idx_appended_to_leader: 1 I0918 20:38:26.487015 20725 consensus_queue.cc:544] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [LEADER]: The tablet replica hosted on peer 644cb35e677e49c3a2210fd6b3f4f8b9 has failed W0918 20:38:26.487547 20725 consensus_peers.cc:458] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e -> Peer 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1:45751): Couldn't send request to peer 644cb35e677e49c3a2210fd6b3f4f8b9. Error code: WRONG_SERVER_UUID (16). Status: Invalid argument: UpdateConsensus: Wrong destination UUID requested. Local UUID: d7a56d8c2b174760bbaefaa16dc0ff8d. Requested UUID: 644cb35e677e49c3a2210fd6b3f4f8b9. This is attempt 1: this message will repeat every 5th retry. I0918 20:38:26.492702 20464 catalog_manager.cc:3882] Sending ChangeConfig:REMOVE_PEER on tablet f7ed278cf9394be89b6ddd9b49a861bb (attempt 1) I0918 20:38:26.497553 20762 consensus_queue.cc:206] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } I0918 20:38:26.501929 20692 raft_consensus.cc:1184] T f7ed278cf9394be89b6ddd9b49a861bb P 413954693a164d5a80637ee06a7bf0b4 [term 1 FOLLOWER]: Refusing update from remote peer f72474f68c3248daad0f6975dd6bea9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:26.503326 20845 consensus_queue.cc:984] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0918 20:38:26.509572 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:26.509811 20878 raft_consensus.cc:2792] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } } I0918 20:38:26.511368 20692 raft_consensus.cc:2792] T f7ed278cf9394be89b6ddd9b49a861bb P 413954693a164d5a80637ee06a7bf0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } } I0918 20:38:26.538319 20447 catalog_manager.cc:3665] ChangeConfig:REMOVE_PEER RPC for tablet f7ed278cf9394be89b6ddd9b49a861bb with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0918 20:38:26.538636 20463 catalog_manager.cc:4131] T f7ed278cf9394be89b6ddd9b49a861bb P 413954693a164d5a80637ee06a7bf0b4 reported cstate change: config changed from index -1 to 2, VOTER 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1) evicted. New cstate: current_term: 1 leader_uuid: "f72474f68c3248daad0f6975dd6bea9e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } } I0918 20:38:26.587481 20463 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet f7ed278cf9394be89b6ddd9b49a861bb on 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1:45751) (TS 644cb35e677e49c3a2210fd6b3f4f8b9 not found in new config with opid_index 2) I0918 20:38:26.589301 20464 catalog_manager.cc:3750] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet f7ed278cf9394be89b6ddd9b49a861bb (attempt 1) I0918 20:38:26.603273 20762 consensus_queue.cc:206] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: NON_VOTER last_known_addr { host: "127.18.222.1" port: 45751 } attrs { promote: true } } I0918 20:38:26.620102 20892 messenger.cc:381] Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer W0918 20:38:26.621212 20725 consensus_peers.cc:458] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e -> Peer 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1:45751): Couldn't send request to peer 644cb35e677e49c3a2210fd6b3f4f8b9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry. I0918 20:38:26.621915 20692 raft_consensus.cc:1184] T f7ed278cf9394be89b6ddd9b49a861bb P 413954693a164d5a80637ee06a7bf0b4 [term 1 FOLLOWER]: Refusing update from remote peer f72474f68c3248daad0f6975dd6bea9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I0918 20:38:26.622404 20464 catalog_manager.cc:3750] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet f7ed278cf9394be89b6ddd9b49a861bb (attempt 1) I0918 20:38:26.623535 20878 consensus_queue.cc:984] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0918 20:38:26.626684 20447 catalog_manager.cc:3682] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f7ed278cf9394be89b6ddd9b49a861bb with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER failed with leader f72474f68c3248daad0f6975dd6bea9e (127.18.222.3:41199) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } }. Pending config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: NON_VOTER last_known_addr { host: "127.18.222.1" port: 45751 } attrs { promote: true } } I0918 20:38:26.627409 20447 catalog_manager.cc:3298] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f7ed278cf9394be89b6ddd9b49a861bb with cas_config_opid_index 2 with a delay of 64 ms (attempt = 1) I0918 20:38:26.630471 20845 raft_consensus.cc:2792] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: NON_VOTER last_known_addr { host: "127.18.222.1" port: 45751 } attrs { promote: true } } } I0918 20:38:26.632951 20692 raft_consensus.cc:2792] T f7ed278cf9394be89b6ddd9b49a861bb P 413954693a164d5a80637ee06a7bf0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } } peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: NON_VOTER last_known_addr { host: "127.18.222.1" port: 45751 } attrs { promote: true } } } I0918 20:38:26.641649 20447 catalog_manager.cc:3665] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f7ed278cf9394be89b6ddd9b49a861bb with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0918 20:38:26.643976 20464 catalog_manager.cc:4131] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e reported cstate change: config changed from index 2 to 3, NON_VOTER 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1) added. New cstate: current_term: 1 leader_uuid: "f72474f68c3248daad0f6975dd6bea9e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "413954693a164d5a80637ee06a7bf0b4" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f72474f68c3248daad0f6975dd6bea9e" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "644cb35e677e49c3a2210fd6b3f4f8b9" member_type: NON_VOTER last_known_addr { host: "127.18.222.1" port: 45751 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0918 20:38:26.654419 20892 messenger.cc:381] Service unavailable: service kudu.tserver.TabletServerAdminService not registered on TabletServer W0918 20:38:26.657487 20447 catalog_manager.cc:3262] TS 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1:45751): Delete Tablet RPC failed for tablet f7ed278cf9394be89b6ddd9b49a861bb: Remote error: Service unavailable: service kudu.tserver.TabletServerAdminService not registered on TabletServer I0918 20:38:26.657829 20447 catalog_manager.cc:3298] Scheduling retry of f7ed278cf9394be89b6ddd9b49a861bb Delete Tablet RPC for TS=644cb35e677e49c3a2210fd6b3f4f8b9 with a delay of 41 ms (attempt = 1) I0918 20:38:26.665496 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:26.677052 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:26.677314 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:26.677686 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:26.685695 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:26.692762 20447 catalog_manager.cc:3698] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f7ed278cf9394be89b6ddd9b49a861bb with cas_config_opid_index 2: aborting the task: latest config opid_index 3; task opid_index 2 I0918 20:38:26.700024 20447 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet f7ed278cf9394be89b6ddd9b49a861bb on 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1:45751) (TS 644cb35e677e49c3a2210fd6b3f4f8b9 not found in new config with opid_index 2) W0918 20:38:26.702630 20447 catalog_manager.cc:3262] TS 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1:45751): Delete Tablet RPC failed for tablet f7ed278cf9394be89b6ddd9b49a861bb: Network error: Client connection negotiation failed: client connection to 127.18.222.1:45751: connect: Connection refused (error 111) I0918 20:38:26.702973 20447 catalog_manager.cc:3298] Scheduling retry of f7ed278cf9394be89b6ddd9b49a861bb Delete Tablet RPC for TS=644cb35e677e49c3a2210fd6b3f4f8b9 with a delay of 56 ms (attempt = 2) I0918 20:38:26.727619 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:26.728008 19320 tablet_replica.cc:273] T f7ed278cf9394be89b6ddd9b49a861bb P 413954693a164d5a80637ee06a7bf0b4: stopping tablet replica I0918 20:38:26.728407 19320 raft_consensus.cc:2147] T f7ed278cf9394be89b6ddd9b49a861bb P 413954693a164d5a80637ee06a7bf0b4 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:26.728761 19320 raft_consensus.cc:2174] T f7ed278cf9394be89b6ddd9b49a861bb P 413954693a164d5a80637ee06a7bf0b4 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:26.732575 19320 tablet_replica.cc:273] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 413954693a164d5a80637ee06a7bf0b4: stopping tablet replica I0918 20:38:26.732934 19320 raft_consensus.cc:2147] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 413954693a164d5a80637ee06a7bf0b4 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:26.733206 19320 raft_consensus.cc:2174] T 51e4b66c13a1478b813c2f9b8af9f1e9 P 413954693a164d5a80637ee06a7bf0b4 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:26.735729 19320 tablet_replica.cc:273] T 199cb07c60bf471a801872c84de7b218 P 413954693a164d5a80637ee06a7bf0b4: stopping tablet replica I0918 20:38:26.736076 19320 raft_consensus.cc:2147] T 199cb07c60bf471a801872c84de7b218 P 413954693a164d5a80637ee06a7bf0b4 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:26.736337 19320 raft_consensus.cc:2174] T 199cb07c60bf471a801872c84de7b218 P 413954693a164d5a80637ee06a7bf0b4 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:26.749599 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:26.749804 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:26.750104 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:26.759760 20447 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet f7ed278cf9394be89b6ddd9b49a861bb on 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1:45751) (TS 644cb35e677e49c3a2210fd6b3f4f8b9 not found in new config with opid_index 2) W0918 20:38:26.761293 20447 catalog_manager.cc:3262] TS 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1:45751): Delete Tablet RPC failed for tablet f7ed278cf9394be89b6ddd9b49a861bb: Network error: Client connection negotiation failed: client connection to 127.18.222.1:45751: connect: Connection refused (error 111) I0918 20:38:26.761605 20447 catalog_manager.cc:3298] Scheduling retry of f7ed278cf9394be89b6ddd9b49a861bb Delete Tablet RPC for TS=644cb35e677e49c3a2210fd6b3f4f8b9 with a delay of 110 ms (attempt = 3) I0918 20:38:26.763118 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:26.791574 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:26.791893 19320 tablet_replica.cc:273] T 51e4b66c13a1478b813c2f9b8af9f1e9 P f72474f68c3248daad0f6975dd6bea9e: stopping tablet replica I0918 20:38:26.792240 19320 raft_consensus.cc:2147] T 51e4b66c13a1478b813c2f9b8af9f1e9 P f72474f68c3248daad0f6975dd6bea9e [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:26.792500 19320 raft_consensus.cc:2174] T 51e4b66c13a1478b813c2f9b8af9f1e9 P f72474f68c3248daad0f6975dd6bea9e [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:26.795169 19320 tablet_replica.cc:273] T 199cb07c60bf471a801872c84de7b218 P f72474f68c3248daad0f6975dd6bea9e: stopping tablet replica I0918 20:38:26.795538 19320 raft_consensus.cc:2147] T 199cb07c60bf471a801872c84de7b218 P f72474f68c3248daad0f6975dd6bea9e [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:26.795833 19320 raft_consensus.cc:2174] T 199cb07c60bf471a801872c84de7b218 P f72474f68c3248daad0f6975dd6bea9e [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:26.797981 19320 tablet_replica.cc:273] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e: stopping tablet replica I0918 20:38:26.798308 19320 raft_consensus.cc:2147] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:26.798928 19320 raft_consensus.cc:2174] T f7ed278cf9394be89b6ddd9b49a861bb P f72474f68c3248daad0f6975dd6bea9e [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:26.814165 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:26.814381 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:26.814743 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:26.824082 19320 master.cc:247] Master@127.18.222.62:41123 shutting down... I0918 20:38:26.840481 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:26.840859 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:26.841068 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 4fc5f353671b4c1d8241bd0dd8e762b5: stopping tablet replica I0918 20:38:26.861030 19320 master.cc:258] Master@127.18.222.62:41123 shutdown complete. I0918 20:38:26.870532 19320 master.cc:247] Master@127.18.222.61:44665 shutting down... I0918 20:38:26.873148 20447 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet f7ed278cf9394be89b6ddd9b49a861bb on 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1:45751) (TS 644cb35e677e49c3a2210fd6b3f4f8b9 not found in new config with opid_index 2) W0918 20:38:26.875084 20447 catalog_manager.cc:3262] TS 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1:45751): Delete Tablet RPC failed for tablet f7ed278cf9394be89b6ddd9b49a861bb: Network error: Client connection negotiation failed: client connection to 127.18.222.1:45751: connect: Connection refused (error 111) I0918 20:38:26.875381 20447 catalog_manager.cc:3298] Scheduling retry of f7ed278cf9394be89b6ddd9b49a861bb Delete Tablet RPC for TS=644cb35e677e49c3a2210fd6b3f4f8b9 with a delay of 148 ms (attempt = 4) I0918 20:38:27.024828 20447 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet f7ed278cf9394be89b6ddd9b49a861bb on 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1:45751) (TS 644cb35e677e49c3a2210fd6b3f4f8b9 not found in new config with opid_index 2) W0918 20:38:27.026458 20447 catalog_manager.cc:3262] TS 644cb35e677e49c3a2210fd6b3f4f8b9 (127.18.222.1:45751): Delete Tablet RPC failed for tablet f7ed278cf9394be89b6ddd9b49a861bb: Network error: Client connection negotiation failed: client connection to 127.18.222.1:45751: connect: Connection refused (error 111) I0918 20:38:27.051808 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:27.052613 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:27.052852 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 6e70059559ab43c7ad0ec1fc928d667e: stopping tablet replica I0918 20:38:27.072820 19320 master.cc:258] Master@127.18.222.61:44665 shutdown complete. I0918 20:38:27.081426 19320 master.cc:247] Master@127.18.222.60:42463 shutting down... I0918 20:38:27.098362 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:27.098729 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:27.098955 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 893da579794d43378162b1bf6704dc54: stopping tablet replica I0918 20:38:27.118463 19320 master.cc:258] Master@127.18.222.60:42463 shutdown complete. [ OK ] RemoteKsckTest.TestTabletServerMismatchedUUID (3578 ms) [ RUN ] RemoteKsckTest.TestTableConsistency I0918 20:38:27.146422 19320 test_util.cc:212] Using random seed: -776932707 I0918 20:38:27.147156 19320 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.18.222.62:32943,127.18.222.61:40493,127.18.222.60:44745 I0918 20:38:27.159451 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 131092us I0918 20:38:27.159821 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:27.159979 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:27.160187 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:27.160316 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:27.161295 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/master-0-root/instance: uuid: "fbf7f3ed055e466888130f93b574c7b5" format_stamp: "Formatted at 2019-09-18 20:38:27 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:27.169399 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.008s sys 0.001s I0918 20:38:27.170791 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:27.177873 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:27.178066 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/master-0-root uuid: "fbf7f3ed055e466888130f93b574c7b5" format_stamp: "Formatted at 2019-09-18 20:38:27 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:27.178323 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:27.256151 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:32943 I0918 20:38:27.257908 19320 webserver.cc:374] Webserver started at http://127.18.222.62:41139/ using document root and password file I0918 20:38:27.260110 21031 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:27.266371 21031 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:27.279083 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 131092us I0918 20:38:27.279595 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:27.279841 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:27.280118 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:27.280318 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:27.281582 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/master-1-root/instance: uuid: "74d1e2b653404613bbdfa0e3c67634b7" format_stamp: "Formatted at 2019-09-18 20:38:27 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:27.284193 21031 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } has no permanent_uuid. Determining permanent_uuid... W0918 20:38:27.285920 21031 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:40493: Network error: Client connection negotiation failed: client connection to 127.18.222.61:40493: connect: Connection refused (error 111) I0918 20:38:27.291421 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.011s sys 0.000s I0918 20:38:27.292985 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:27.301116 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:27.301322 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/master-1-root uuid: "74d1e2b653404613bbdfa0e3c67634b7" format_stamp: "Formatted at 2019-09-18 20:38:27 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:27.301580 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:27.335340 21031 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } attempt: 1 W0918 20:38:27.337725 21031 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:40493: Network error: Client connection negotiation failed: client connection to 127.18.222.61:40493: connect: Connection refused (error 111) I0918 20:38:27.348989 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:40493 I0918 20:38:27.350775 19320 webserver.cc:374] Webserver started at http://127.18.222.61:37193/ using document root and password file I0918 20:38:27.352596 21088 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:27.356935 21088 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:27.369594 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 131092us I0918 20:38:27.369956 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:27.370172 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:27.370434 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:27.370339 21088 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:27.370632 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:27.371812 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/master-2-root/instance: uuid: "2362c3ddd9814f6b9cf972f09ea443fb" format_stamp: "Formatted at 2019-09-18 20:38:27 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:27.386646 21088 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:27.387889 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.015s user 0.013s sys 0.000s W0918 20:38:27.388664 21088 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:44745: Network error: Client connection negotiation failed: client connection to 127.18.222.60:44745: connect: Connection refused (error 111) I0918 20:38:27.389696 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:27.397598 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:27.397809 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/master-2-root uuid: "2362c3ddd9814f6b9cf972f09ea443fb" format_stamp: "Formatted at 2019-09-18 20:38:27 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:27.398064 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:27.405184 21031 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } attempt: 2 I0918 20:38:27.416313 21031 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } has no permanent_uuid. Determining permanent_uuid... W0918 20:38:27.418049 21031 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:44745: Network error: Client connection negotiation failed: client connection to 127.18.222.60:44745: connect: Connection refused (error 111) I0918 20:38:27.441037 21088 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } attempt: 1 W0918 20:38:27.443310 21088 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:44745: Network error: Client connection negotiation failed: client connection to 127.18.222.60:44745: connect: Connection refused (error 111) I0918 20:38:27.456468 21031 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } attempt: 1 I0918 20:38:27.457152 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:44745 I0918 20:38:27.460203 19320 webserver.cc:374] Webserver started at http://127.18.222.60:44389/ using document root and password file I0918 20:38:27.461824 21146 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:27.464818 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0918 20:38:27.466789 21146 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:27.472149 21031 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5: Bootstrap starting. I0918 20:38:27.474678 21031 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5: No blocks or log segments found. Creating new log. I0918 20:38:27.478241 21031 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5: No bootstrap required, opened a new log I0918 20:38:27.478631 21146 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:27.480213 21031 raft_consensus.cc:340] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } I0918 20:38:27.480911 21031 raft_consensus.cc:366] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:27.481103 21031 raft_consensus.cc:705] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbf7f3ed055e466888130f93b574c7b5, State: Initialized, Role: FOLLOWER I0918 20:38:27.481782 21031 consensus_queue.cc:229] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } I0918 20:38:27.483675 21149 sys_catalog.cc:340] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } } I0918 20:38:27.484283 21149 sys_catalog.cc:343] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:27.483532 21031 sys_catalog.cc:340] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } } I0918 20:38:27.484645 21031 sys_catalog.cc:343] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:27.484664 21088 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } attempt: 2 I0918 20:38:27.485708 21031 sys_catalog.cc:428] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:38:27.486904 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 W0918 20:38:27.489632 21150 catalog_manager.cc:1155] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:38:27.489969 21150 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:27.492110 21146 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:27.502683 21088 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7: Bootstrap starting. I0918 20:38:27.506183 21088 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7: No blocks or log segments found. Creating new log. I0918 20:38:27.509171 21088 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7: No bootstrap required, opened a new log I0918 20:38:27.510097 21149 raft_consensus.cc:472] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:27.510736 21088 raft_consensus.cc:340] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } I0918 20:38:27.510555 21149 raft_consensus.cc:494] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } I0918 20:38:27.511278 21088 raft_consensus.cc:366] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:27.511504 21088 raft_consensus.cc:705] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d1e2b653404613bbdfa0e3c67634b7, State: Initialized, Role: FOLLOWER I0918 20:38:27.512471 21146 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb: Bootstrap starting. I0918 20:38:27.512054 21088 consensus_queue.cc:229] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } I0918 20:38:27.512979 21149 leader_election.cc:296] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74d1e2b653404613bbdfa0e3c67634b7 (127.18.222.61:40493), 2362c3ddd9814f6b9cf972f09ea443fb (127.18.222.60:44745) I0918 20:38:27.514443 21088 sys_catalog.cc:340] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } } I0918 20:38:27.514736 21153 sys_catalog.cc:340] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } } I0918 20:38:27.515326 21088 sys_catalog.cc:343] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:27.515622 21153 sys_catalog.cc:343] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:27.516685 21146 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb: No blocks or log segments found. Creating new log. I0918 20:38:27.516728 21088 sys_catalog.cc:428] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:38:27.516799 21065 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbf7f3ed055e466888130f93b574c7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d1e2b653404613bbdfa0e3c67634b7" is_pre_election: true I0918 20:38:27.517733 21065 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbf7f3ed055e466888130f93b574c7b5 in term 0. I0918 20:38:27.519034 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 I0918 20:38:27.519147 20979 leader_election.cc:310] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74d1e2b653404613bbdfa0e3c67634b7, fbf7f3ed055e466888130f93b574c7b5; no voters: I0918 20:38:27.520210 21149 raft_consensus.cc:2644] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:27.520512 21149 raft_consensus.cc:472] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:27.520834 21149 raft_consensus.cc:2890] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:27.522210 21146 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb: No bootstrap required, opened a new log W0918 20:38:27.522241 21154 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:38:27.523897 21154 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:27.524924 21146 raft_consensus.cc:340] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } I0918 20:38:27.525719 21146 raft_consensus.cc:366] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:27.525630 21149 raft_consensus.cc:494] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } I0918 20:38:27.526026 21146 raft_consensus.cc:705] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2362c3ddd9814f6b9cf972f09ea443fb, State: Initialized, Role: FOLLOWER I0918 20:38:27.526760 21146 consensus_queue.cc:229] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } I0918 20:38:27.527449 21149 leader_election.cc:296] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [CANDIDATE]: Term 1 election: Requested vote from peers 74d1e2b653404613bbdfa0e3c67634b7 (127.18.222.61:40493), 2362c3ddd9814f6b9cf972f09ea443fb (127.18.222.60:44745) I0918 20:38:27.528043 21065 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbf7f3ed055e466888130f93b574c7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d1e2b653404613bbdfa0e3c67634b7" I0918 20:38:27.528821 21065 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:27.528836 21146 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } } I0918 20:38:27.529115 21156 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } } I0918 20:38:27.529773 21146 sys_catalog.cc:343] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:27.529927 21156 sys_catalog.cc:343] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:27.531059 21146 sys_catalog.cc:428] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb [sys.catalog]: configured and running, proceeding with master startup. I0918 20:38:27.531111 21122 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbf7f3ed055e466888130f93b574c7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" is_pre_election: true I0918 20:38:27.532021 21122 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbf7f3ed055e466888130f93b574c7b5 in term 0. I0918 20:38:27.531170 21121 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbf7f3ed055e466888130f93b574c7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" I0918 20:38:27.533681 21121 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:27.534868 21065 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbf7f3ed055e466888130f93b574c7b5 in term 1. I0918 20:38:27.536237 20979 leader_election.cc:310] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74d1e2b653404613bbdfa0e3c67634b7, fbf7f3ed055e466888130f93b574c7b5; no voters: I0918 20:38:27.537221 21149 raft_consensus.cc:2644] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:27.538709 21121 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbf7f3ed055e466888130f93b574c7b5 in term 1. I0918 20:38:27.540057 21149 raft_consensus.cc:667] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [term 1 LEADER]: Becoming Leader. State: Replica: fbf7f3ed055e466888130f93b574c7b5, State: Running, Role: LEADER I0918 20:38:27.541031 21149 consensus_queue.cc:206] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } W0918 20:38:27.543996 21157 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:38:27.544360 21157 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:27.546008 21158 sys_catalog.cc:340] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbf7f3ed055e466888130f93b574c7b5. Latest consensus state: current_term: 1 leader_uuid: "fbf7f3ed055e466888130f93b574c7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } } I0918 20:38:27.546818 21158 sys_catalog.cc:343] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [sys.catalog]: This master's current role is: LEADER I0918 20:38:27.548359 21160 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:38:27.550669 21160 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:38:27.556404 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 131092us I0918 20:38:27.556754 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:27.556922 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:27.557142 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:27.557284 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:27.558290 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/ts-0-root/instance: uuid: "3ff36990b0c44cf7b62f9e2df4a80087" format_stamp: "Formatted at 2019-09-18 20:38:27 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:27.569068 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.009s sys 0.004s I0918 20:38:27.570713 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:27.580515 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:27.580744 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/ts-0-root uuid: "3ff36990b0c44cf7b62f9e2df4a80087" format_stamp: "Formatted at 2019-09-18 20:38:27 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:27.581007 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:27.605957 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:27.606230 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:27.607133 21121 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb [term 1 FOLLOWER]: Refusing update from remote peer fbf7f3ed055e466888130f93b574c7b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:27.607393 21065 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7 [term 1 FOLLOWER]: Refusing update from remote peer fbf7f3ed055e466888130f93b574c7b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:27.608729 21149 consensus_queue.cc:984] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:27.609372 21158 consensus_queue.cc:984] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:27.617374 21156 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbf7f3ed055e466888130f93b574c7b5. Latest consensus state: current_term: 1 leader_uuid: "fbf7f3ed055e466888130f93b574c7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } } I0918 20:38:27.617971 21156 sys_catalog.cc:343] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:27.621023 21153 sys_catalog.cc:340] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbf7f3ed055e466888130f93b574c7b5. Latest consensus state: current_term: 1 leader_uuid: "fbf7f3ed055e466888130f93b574c7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } } I0918 20:38:27.622359 21153 sys_catalog.cc:343] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:27.635236 21149 sys_catalog.cc:340] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fbf7f3ed055e466888130f93b574c7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } } I0918 20:38:27.636426 21149 sys_catalog.cc:343] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [sys.catalog]: This master's current role is: LEADER I0918 20:38:27.637454 21149 sys_catalog.cc:340] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fbf7f3ed055e466888130f93b574c7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } } I0918 20:38:27.639497 21149 sys_catalog.cc:343] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [sys.catalog]: This master's current role is: LEADER I0918 20:38:27.640858 21160 catalog_manager.cc:981] Generated new certificate authority record I0918 20:38:27.642853 21153 sys_catalog.cc:340] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fbf7f3ed055e466888130f93b574c7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } } I0918 20:38:27.643610 21153 sys_catalog.cc:343] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:27.644582 21160 catalog_manager.cc:1106] Loading token signing keys... I0918 20:38:27.645331 21156 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fbf7f3ed055e466888130f93b574c7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf7f3ed055e466888130f93b574c7b5" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 32943 } } peers { permanent_uuid: "74d1e2b653404613bbdfa0e3c67634b7" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40493 } } peers { permanent_uuid: "2362c3ddd9814f6b9cf972f09ea443fb" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44745 } } } I0918 20:38:27.646121 21156 sys_catalog.cc:343] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:27.664762 21160 catalog_manager.cc:4446] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5: Generated new TSK 0 I0918 20:38:27.665539 21160 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:38:27.686890 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:41435 I0918 20:38:27.688776 19320 webserver.cc:374] Webserver started at http://127.18.222.1:38135/ using document root and password file I0918 20:38:27.713979 21235 heartbeater.cc:322] Connected to a master server at 127.18.222.61:40493 I0918 20:38:27.714361 21235 heartbeater.cc:406] Registering TS with master... I0918 20:38:27.714941 21235 heartbeater.cc:455] Master 127.18.222.61:40493 requested a full tablet report, sending... I0918 20:38:27.715274 21234 heartbeater.cc:322] Connected to a master server at 127.18.222.60:44745 I0918 20:38:27.715656 21234 heartbeater.cc:406] Registering TS with master... I0918 20:38:27.716193 21234 heartbeater.cc:455] Master 127.18.222.60:44745 requested a full tablet report, sending... I0918 20:38:27.717514 21055 ts_manager.cc:186] Registered new tserver with Master: 3ff36990b0c44cf7b62f9e2df4a80087 (127.18.222.1:41435) I0918 20:38:27.718747 21112 ts_manager.cc:186] Registered new tserver with Master: 3ff36990b0c44cf7b62f9e2df4a80087 (127.18.222.1:41435) I0918 20:38:27.720129 21236 heartbeater.cc:322] Connected to a master server at 127.18.222.62:32943 I0918 20:38:27.720469 21236 heartbeater.cc:406] Registering TS with master... I0918 20:38:27.720886 21236 heartbeater.cc:455] Master 127.18.222.62:32943 requested a full tablet report, sending... I0918 20:38:27.721959 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 131092us I0918 20:38:27.722455 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:27.722592 20998 ts_manager.cc:186] Registered new tserver with Master: 3ff36990b0c44cf7b62f9e2df4a80087 (127.18.222.1:41435) I0918 20:38:27.722699 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:27.723083 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:27.723289 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:27.724684 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/ts-1-root/instance: uuid: "1514f228147341d5baafc932fda5d90f" format_stamp: "Formatted at 2019-09-18 20:38:27 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:27.725540 20998 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43550 I0918 20:38:27.736964 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.012s user 0.011s sys 0.001s I0918 20:38:27.738226 21236 heartbeater.cc:447] Master 127.18.222.62:32943 was elected leader, sending a full tablet report... I0918 20:38:27.738512 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:27.748183 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:27.748383 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/ts-1-root uuid: "1514f228147341d5baafc932fda5d90f" format_stamp: "Formatted at 2019-09-18 20:38:27 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:27.748638 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:27.784866 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:27.785184 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:27.833714 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.2:41977 I0918 20:38:27.835670 19320 webserver.cc:374] Webserver started at http://127.18.222.2:33023/ using document root and password file I0918 20:38:27.858515 21305 heartbeater.cc:322] Connected to a master server at 127.18.222.60:44745 I0918 20:38:27.858871 21305 heartbeater.cc:406] Registering TS with master... I0918 20:38:27.859483 21305 heartbeater.cc:455] Master 127.18.222.60:44745 requested a full tablet report, sending... I0918 20:38:27.862149 21112 ts_manager.cc:186] Registered new tserver with Master: 1514f228147341d5baafc932fda5d90f (127.18.222.2:41977) I0918 20:38:27.867009 21306 heartbeater.cc:322] Connected to a master server at 127.18.222.61:40493 I0918 20:38:27.867305 21306 heartbeater.cc:406] Registering TS with master... I0918 20:38:27.867786 21306 heartbeater.cc:455] Master 127.18.222.61:40493 requested a full tablet report, sending... I0918 20:38:27.870062 21054 ts_manager.cc:186] Registered new tserver with Master: 1514f228147341d5baafc932fda5d90f (127.18.222.2:41977) I0918 20:38:27.872565 21308 heartbeater.cc:322] Connected to a master server at 127.18.222.62:32943 I0918 20:38:27.872882 21308 heartbeater.cc:406] Registering TS with master... I0918 20:38:27.873461 21308 heartbeater.cc:455] Master 127.18.222.62:32943 requested a full tablet report, sending... I0918 20:38:27.874575 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 131092us I0918 20:38:27.875047 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:27.875286 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:27.875283 20998 ts_manager.cc:186] Registered new tserver with Master: 1514f228147341d5baafc932fda5d90f (127.18.222.2:41977) I0918 20:38:27.875649 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:27.875929 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:27.877250 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/ts-2-root/instance: uuid: "862f8a414df940f490cbe38227675ef4" format_stamp: "Formatted at 2019-09-18 20:38:27 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:27.877565 20998 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43556 I0918 20:38:27.890134 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.012s user 0.013s sys 0.000s I0918 20:38:27.890693 21308 heartbeater.cc:447] Master 127.18.222.62:32943 was elected leader, sending a full tablet report... I0918 20:38:27.892048 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:27.902474 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0918 20:38:27.902698 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/ts-2-root uuid: "862f8a414df940f490cbe38227675ef4" format_stamp: "Formatted at 2019-09-18 20:38:27 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:27.902976 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1568839089958813-19320/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:27.956301 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:27.956513 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:28.008368 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.3:45615 I0918 20:38:28.010541 19320 webserver.cc:374] Webserver started at http://127.18.222.3:34373/ using document root and password file I0918 20:38:28.101837 21376 heartbeater.cc:322] Connected to a master server at 127.18.222.60:44745 I0918 20:38:28.102214 21376 heartbeater.cc:406] Registering TS with master... I0918 20:38:28.102957 21376 heartbeater.cc:455] Master 127.18.222.60:44745 requested a full tablet report, sending... I0918 20:38:28.105939 21111 ts_manager.cc:186] Registered new tserver with Master: 862f8a414df940f490cbe38227675ef4 (127.18.222.3:45615) I0918 20:38:28.110419 21377 heartbeater.cc:322] Connected to a master server at 127.18.222.61:40493 I0918 20:38:28.110795 21377 heartbeater.cc:406] Registering TS with master... I0918 20:38:28.111397 21377 heartbeater.cc:455] Master 127.18.222.61:40493 requested a full tablet report, sending... I0918 20:38:28.113972 21054 ts_manager.cc:186] Registered new tserver with Master: 862f8a414df940f490cbe38227675ef4 (127.18.222.3:45615) I0918 20:38:28.117766 21378 heartbeater.cc:322] Connected to a master server at 127.18.222.62:32943 I0918 20:38:28.118118 21378 heartbeater.cc:406] Registering TS with master... I0918 20:38:28.118598 21378 heartbeater.cc:455] Master 127.18.222.62:32943 requested a full tablet report, sending... I0918 20:38:28.120604 20998 ts_manager.cc:186] Registered new tserver with Master: 862f8a414df940f490cbe38227675ef4 (127.18.222.3:45615) I0918 20:38:28.120791 19320 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.037498722s I0918 20:38:28.123407 20998 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43562 I0918 20:38:28.142038 21378 heartbeater.cc:447] Master 127.18.222.62:32943 was elected leader, sending a full tablet report... I0918 20:38:28.218905 20996 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43568: name: "ksck-test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\n\000\000\000\004\001\024\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:38:28.220794 20996 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:38:28.385399 21343 tablet_service.cc:1022] Processing CreateTablet for tablet 1a31b5a965a7404bbda0d2779e584d53 (table=ksck-test-table [id=7553ef1a3cbd4109ae466d89baed73f8]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:28.386669 21343 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 1a31b5a965a7404bbda0d2779e584d53. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:28.387729 21342 tablet_service.cc:1022] Processing CreateTablet for tablet 17a83b80ba4e44bb979907ef18954b69 (table=ksck-test-table [id=7553ef1a3cbd4109ae466d89baed73f8]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:28.388198 21341 tablet_service.cc:1022] Processing CreateTablet for tablet 542bf88d39f1442d94d8d8e9084447f7 (table=ksck-test-table [id=7553ef1a3cbd4109ae466d89baed73f8]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:28.388747 21342 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 17a83b80ba4e44bb979907ef18954b69. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:28.389410 21341 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 542bf88d39f1442d94d8d8e9084447f7. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:28.405990 21405 tablet_bootstrap.cc:439] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4: Bootstrap starting. I0918 20:38:28.409642 21405 tablet_bootstrap.cc:590] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4: No blocks or log segments found. Creating new log. I0918 20:38:28.422523 21405 tablet_bootstrap.cc:439] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4: No bootstrap required, opened a new log I0918 20:38:28.423012 21405 ts_tablet_manager.cc:1085] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.003s I0918 20:38:28.425186 21405 raft_consensus.cc:340] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } I0918 20:38:28.425943 21405 raft_consensus.cc:366] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:28.426163 21405 raft_consensus.cc:705] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862f8a414df940f490cbe38227675ef4, State: Initialized, Role: FOLLOWER I0918 20:38:28.426986 21405 consensus_queue.cc:229] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } I0918 20:38:28.429085 21405 ts_tablet_manager.cc:1113] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s I0918 20:38:28.429639 21405 tablet_bootstrap.cc:439] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4: Bootstrap starting. I0918 20:38:28.432139 21405 tablet_bootstrap.cc:590] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4: No blocks or log segments found. Creating new log. I0918 20:38:28.436776 21405 tablet_bootstrap.cc:439] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4: No bootstrap required, opened a new log I0918 20:38:28.440229 21201 tablet_service.cc:1022] Processing CreateTablet for tablet 1a31b5a965a7404bbda0d2779e584d53 (table=ksck-test-table [id=7553ef1a3cbd4109ae466d89baed73f8]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:28.440356 21200 tablet_service.cc:1022] Processing CreateTablet for tablet 17a83b80ba4e44bb979907ef18954b69 (table=ksck-test-table [id=7553ef1a3cbd4109ae466d89baed73f8]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:28.441361 21199 tablet_service.cc:1022] Processing CreateTablet for tablet 542bf88d39f1442d94d8d8e9084447f7 (table=ksck-test-table [id=7553ef1a3cbd4109ae466d89baed73f8]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:28.441409 21201 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 1a31b5a965a7404bbda0d2779e584d53. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:28.445089 21200 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 17a83b80ba4e44bb979907ef18954b69. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:28.448971 21199 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 542bf88d39f1442d94d8d8e9084447f7. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:28.449568 21405 ts_tablet_manager.cc:1085] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.000s I0918 20:38:28.455327 21405 raft_consensus.cc:340] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } I0918 20:38:28.456135 21405 raft_consensus.cc:366] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:28.456396 21405 raft_consensus.cc:705] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862f8a414df940f490cbe38227675ef4, State: Initialized, Role: FOLLOWER I0918 20:38:28.457129 21405 consensus_queue.cc:229] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } I0918 20:38:28.458441 21272 tablet_service.cc:1022] Processing CreateTablet for tablet 1a31b5a965a7404bbda0d2779e584d53 (table=ksck-test-table [id=7553ef1a3cbd4109ae466d89baed73f8]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:28.459462 21272 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 1a31b5a965a7404bbda0d2779e584d53. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:28.460320 21271 tablet_service.cc:1022] Processing CreateTablet for tablet 17a83b80ba4e44bb979907ef18954b69 (table=ksck-test-table [id=7553ef1a3cbd4109ae466d89baed73f8]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:28.461402 21271 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 17a83b80ba4e44bb979907ef18954b69. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:28.462460 21270 tablet_service.cc:1022] Processing CreateTablet for tablet 542bf88d39f1442d94d8d8e9084447f7 (table=ksck-test-table [id=7553ef1a3cbd4109ae466d89baed73f8]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:28.463582 21270 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 542bf88d39f1442d94d8d8e9084447f7. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:28.469954 21405 ts_tablet_manager.cc:1113] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4: Time spent starting tablet: real 0.020s user 0.002s sys 0.003s I0918 20:38:28.470863 21405 tablet_bootstrap.cc:439] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4: Bootstrap starting. I0918 20:38:28.476259 21409 tablet_bootstrap.cc:439] T 1a31b5a965a7404bbda0d2779e584d53 P 3ff36990b0c44cf7b62f9e2df4a80087: Bootstrap starting. I0918 20:38:28.485960 21409 tablet_bootstrap.cc:590] T 1a31b5a965a7404bbda0d2779e584d53 P 3ff36990b0c44cf7b62f9e2df4a80087: No blocks or log segments found. Creating new log. I0918 20:38:28.490936 21405 tablet_bootstrap.cc:590] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4: No blocks or log segments found. Creating new log. I0918 20:38:28.529476 21411 tablet_bootstrap.cc:439] T 17a83b80ba4e44bb979907ef18954b69 P 1514f228147341d5baafc932fda5d90f: Bootstrap starting. I0918 20:38:28.538866 21154 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7: acquiring CA information for follower catalog manager: success I0918 20:38:28.540846 21409 tablet_bootstrap.cc:439] T 1a31b5a965a7404bbda0d2779e584d53 P 3ff36990b0c44cf7b62f9e2df4a80087: No bootstrap required, opened a new log I0918 20:38:28.542517 21409 ts_tablet_manager.cc:1085] T 1a31b5a965a7404bbda0d2779e584d53 P 3ff36990b0c44cf7b62f9e2df4a80087: Time spent bootstrapping tablet: real 0.070s user 0.010s sys 0.000s I0918 20:38:28.544085 21411 tablet_bootstrap.cc:590] T 17a83b80ba4e44bb979907ef18954b69 P 1514f228147341d5baafc932fda5d90f: No blocks or log segments found. Creating new log. I0918 20:38:28.544342 21407 raft_consensus.cc:472] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:28.544811 21407 raft_consensus.cc:494] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } I0918 20:38:28.546870 21407 leader_election.cc:296] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ff36990b0c44cf7b62f9e2df4a80087 (127.18.222.1:41435), 1514f228147341d5baafc932fda5d90f (127.18.222.2:41977) I0918 20:38:28.547075 21409 raft_consensus.cc:340] T 1a31b5a965a7404bbda0d2779e584d53 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } I0918 20:38:28.548056 21409 raft_consensus.cc:366] T 1a31b5a965a7404bbda0d2779e584d53 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:28.548401 21409 raft_consensus.cc:705] T 1a31b5a965a7404bbda0d2779e584d53 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ff36990b0c44cf7b62f9e2df4a80087, State: Initialized, Role: FOLLOWER I0918 20:38:28.549121 21409 consensus_queue.cc:229] T 1a31b5a965a7404bbda0d2779e584d53 P 3ff36990b0c44cf7b62f9e2df4a80087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } I0918 20:38:28.558542 21405 tablet_bootstrap.cc:439] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4: No bootstrap required, opened a new log I0918 20:38:28.559113 21405 ts_tablet_manager.cc:1085] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4: Time spent bootstrapping tablet: real 0.088s user 0.014s sys 0.011s I0918 20:38:28.561410 21154 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:38:28.564363 21409 ts_tablet_manager.cc:1113] T 1a31b5a965a7404bbda0d2779e584d53 P 3ff36990b0c44cf7b62f9e2df4a80087: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s I0918 20:38:28.565167 21409 tablet_bootstrap.cc:439] T 542bf88d39f1442d94d8d8e9084447f7 P 3ff36990b0c44cf7b62f9e2df4a80087: Bootstrap starting. I0918 20:38:28.568789 21405 raft_consensus.cc:340] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } I0918 20:38:28.569533 21405 raft_consensus.cc:366] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:28.569778 21405 raft_consensus.cc:705] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862f8a414df940f490cbe38227675ef4, State: Initialized, Role: FOLLOWER I0918 20:38:28.571267 21405 consensus_queue.cc:229] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } I0918 20:38:28.574326 21409 tablet_bootstrap.cc:590] T 542bf88d39f1442d94d8d8e9084447f7 P 3ff36990b0c44cf7b62f9e2df4a80087: No blocks or log segments found. Creating new log. I0918 20:38:28.579798 21407 raft_consensus.cc:472] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:28.581055 21407 raft_consensus.cc:494] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } I0918 20:38:28.584969 21407 leader_election.cc:296] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1514f228147341d5baafc932fda5d90f (127.18.222.2:41977), 3ff36990b0c44cf7b62f9e2df4a80087 (127.18.222.1:41435) I0918 20:38:28.585902 21405 ts_tablet_manager.cc:1113] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s I0918 20:38:28.583575 21157 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb: acquiring CA information for follower catalog manager: success I0918 20:38:28.593035 21157 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:38:28.609405 21411 tablet_bootstrap.cc:439] T 17a83b80ba4e44bb979907ef18954b69 P 1514f228147341d5baafc932fda5d90f: No bootstrap required, opened a new log I0918 20:38:28.610050 21411 ts_tablet_manager.cc:1085] T 17a83b80ba4e44bb979907ef18954b69 P 1514f228147341d5baafc932fda5d90f: Time spent bootstrapping tablet: real 0.081s user 0.014s sys 0.015s I0918 20:38:28.611500 21409 tablet_bootstrap.cc:439] T 542bf88d39f1442d94d8d8e9084447f7 P 3ff36990b0c44cf7b62f9e2df4a80087: No bootstrap required, opened a new log I0918 20:38:28.612066 21409 ts_tablet_manager.cc:1085] T 542bf88d39f1442d94d8d8e9084447f7 P 3ff36990b0c44cf7b62f9e2df4a80087: Time spent bootstrapping tablet: real 0.047s user 0.001s sys 0.006s I0918 20:38:28.612385 21411 raft_consensus.cc:340] T 17a83b80ba4e44bb979907ef18954b69 P 1514f228147341d5baafc932fda5d90f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } I0918 20:38:28.613241 21411 raft_consensus.cc:366] T 17a83b80ba4e44bb979907ef18954b69 P 1514f228147341d5baafc932fda5d90f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:28.613479 21411 raft_consensus.cc:705] T 17a83b80ba4e44bb979907ef18954b69 P 1514f228147341d5baafc932fda5d90f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1514f228147341d5baafc932fda5d90f, State: Initialized, Role: FOLLOWER I0918 20:38:28.614135 21411 consensus_queue.cc:229] T 17a83b80ba4e44bb979907ef18954b69 P 1514f228147341d5baafc932fda5d90f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } I0918 20:38:28.614151 21409 raft_consensus.cc:340] T 542bf88d39f1442d94d8d8e9084447f7 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } I0918 20:38:28.615999 21409 raft_consensus.cc:366] T 542bf88d39f1442d94d8d8e9084447f7 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:28.616266 21409 raft_consensus.cc:705] T 542bf88d39f1442d94d8d8e9084447f7 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ff36990b0c44cf7b62f9e2df4a80087, State: Initialized, Role: FOLLOWER I0918 20:38:28.616897 21409 consensus_queue.cc:229] T 542bf88d39f1442d94d8d8e9084447f7 P 3ff36990b0c44cf7b62f9e2df4a80087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } I0918 20:38:28.618171 21210 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "1a31b5a965a7404bbda0d2779e584d53" candidate_uuid: "862f8a414df940f490cbe38227675ef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" is_pre_election: true I0918 20:38:28.618897 21210 raft_consensus.cc:2350] T 1a31b5a965a7404bbda0d2779e584d53 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862f8a414df940f490cbe38227675ef4 in term 0. I0918 20:38:28.619715 21409 ts_tablet_manager.cc:1113] T 542bf88d39f1442d94d8d8e9084447f7 P 3ff36990b0c44cf7b62f9e2df4a80087: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s I0918 20:38:28.620151 21316 leader_election.cc:310] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ff36990b0c44cf7b62f9e2df4a80087, 862f8a414df940f490cbe38227675ef4; no voters: I0918 20:38:28.621117 21407 raft_consensus.cc:2644] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:28.621420 21407 raft_consensus.cc:472] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:28.621711 21407 raft_consensus.cc:2890] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:28.625053 21409 tablet_bootstrap.cc:439] T 17a83b80ba4e44bb979907ef18954b69 P 3ff36990b0c44cf7b62f9e2df4a80087: Bootstrap starting. I0918 20:38:28.617729 21211 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "17a83b80ba4e44bb979907ef18954b69" candidate_uuid: "862f8a414df940f490cbe38227675ef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" is_pre_election: true I0918 20:38:28.629451 21409 tablet_bootstrap.cc:590] T 17a83b80ba4e44bb979907ef18954b69 P 3ff36990b0c44cf7b62f9e2df4a80087: No blocks or log segments found. Creating new log. W0918 20:38:28.631044 21316 leader_election.cc:348] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3ff36990b0c44cf7b62f9e2df4a80087 (127.18.222.1:41435): Illegal state: must be running to vote when last-logged opid is not known I0918 20:38:28.642992 21407 raft_consensus.cc:494] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } I0918 20:38:28.645941 21409 tablet_bootstrap.cc:439] T 17a83b80ba4e44bb979907ef18954b69 P 3ff36990b0c44cf7b62f9e2df4a80087: No bootstrap required, opened a new log I0918 20:38:28.646193 21211 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "1a31b5a965a7404bbda0d2779e584d53" candidate_uuid: "862f8a414df940f490cbe38227675ef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" I0918 20:38:28.646570 21409 ts_tablet_manager.cc:1085] T 17a83b80ba4e44bb979907ef18954b69 P 3ff36990b0c44cf7b62f9e2df4a80087: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.012s I0918 20:38:28.646960 21211 raft_consensus.cc:2890] T 1a31b5a965a7404bbda0d2779e584d53 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:28.649163 21409 raft_consensus.cc:340] T 17a83b80ba4e44bb979907ef18954b69 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } I0918 20:38:28.650101 21409 raft_consensus.cc:366] T 17a83b80ba4e44bb979907ef18954b69 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:28.656320 21282 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "17a83b80ba4e44bb979907ef18954b69" candidate_uuid: "862f8a414df940f490cbe38227675ef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1514f228147341d5baafc932fda5d90f" is_pre_election: true I0918 20:38:28.650406 21409 raft_consensus.cc:705] T 17a83b80ba4e44bb979907ef18954b69 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ff36990b0c44cf7b62f9e2df4a80087, State: Initialized, Role: FOLLOWER I0918 20:38:28.657090 21282 raft_consensus.cc:2350] T 17a83b80ba4e44bb979907ef18954b69 P 1514f228147341d5baafc932fda5d90f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862f8a414df940f490cbe38227675ef4 in term 0. I0918 20:38:28.657649 21281 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "1a31b5a965a7404bbda0d2779e584d53" candidate_uuid: "862f8a414df940f490cbe38227675ef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1514f228147341d5baafc932fda5d90f" is_pre_election: true I0918 20:38:28.658493 21316 leader_election.cc:310] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1514f228147341d5baafc932fda5d90f, 862f8a414df940f490cbe38227675ef4; no voters: 3ff36990b0c44cf7b62f9e2df4a80087 I0918 20:38:28.658942 21280 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "1a31b5a965a7404bbda0d2779e584d53" candidate_uuid: "862f8a414df940f490cbe38227675ef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1514f228147341d5baafc932fda5d90f" I0918 20:38:28.662714 21211 raft_consensus.cc:2350] T 1a31b5a965a7404bbda0d2779e584d53 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862f8a414df940f490cbe38227675ef4 in term 1. I0918 20:38:28.644944 21407 leader_election.cc:296] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [CANDIDATE]: Term 1 election: Requested vote from peers 1514f228147341d5baafc932fda5d90f (127.18.222.2:41977), 3ff36990b0c44cf7b62f9e2df4a80087 (127.18.222.1:41435) I0918 20:38:28.670653 21407 raft_consensus.cc:2644] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:28.670958 21407 raft_consensus.cc:472] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:28.671257 21407 raft_consensus.cc:2890] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:28.673532 21316 leader_election.cc:310] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ff36990b0c44cf7b62f9e2df4a80087, 862f8a414df940f490cbe38227675ef4; no voters: I0918 20:38:28.675774 21411 ts_tablet_manager.cc:1113] T 17a83b80ba4e44bb979907ef18954b69 P 1514f228147341d5baafc932fda5d90f: Time spent starting tablet: real 0.065s user 0.027s sys 0.035s I0918 20:38:28.676342 21409 consensus_queue.cc:229] T 17a83b80ba4e44bb979907ef18954b69 P 3ff36990b0c44cf7b62f9e2df4a80087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } I0918 20:38:28.684082 21418 raft_consensus.cc:472] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:28.684654 21418 raft_consensus.cc:494] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } I0918 20:38:28.686206 21418 leader_election.cc:296] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1514f228147341d5baafc932fda5d90f (127.18.222.2:41977), 3ff36990b0c44cf7b62f9e2df4a80087 (127.18.222.1:41435) I0918 20:38:28.687355 21418 raft_consensus.cc:2644] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:28.692554 21407 raft_consensus.cc:494] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } I0918 20:38:28.704056 21418 raft_consensus.cc:667] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [term 1 LEADER]: Becoming Leader. State: Replica: 862f8a414df940f490cbe38227675ef4, State: Running, Role: LEADER W0918 20:38:28.704344 21316 leader_election.cc:348] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1514f228147341d5baafc932fda5d90f (127.18.222.2:41977): Illegal state: must be running to vote when last-logged opid is not known I0918 20:38:28.705061 21407 leader_election.cc:296] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [CANDIDATE]: Term 1 election: Requested vote from peers 3ff36990b0c44cf7b62f9e2df4a80087 (127.18.222.1:41435), 1514f228147341d5baafc932fda5d90f (127.18.222.2:41977) I0918 20:38:28.704823 21418 consensus_queue.cc:206] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } I0918 20:38:28.694522 21409 ts_tablet_manager.cc:1113] T 17a83b80ba4e44bb979907ef18954b69 P 3ff36990b0c44cf7b62f9e2df4a80087: Time spent starting tablet: real 0.048s user 0.005s sys 0.000s I0918 20:38:28.699012 21411 tablet_bootstrap.cc:439] T 1a31b5a965a7404bbda0d2779e584d53 P 1514f228147341d5baafc932fda5d90f: Bootstrap starting. W0918 20:38:28.706300 21316 leader_election.cc:348] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1514f228147341d5baafc932fda5d90f (127.18.222.2:41977): Illegal state: must be running to vote when last-logged opid is not known I0918 20:38:28.707670 21211 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "542bf88d39f1442d94d8d8e9084447f7" candidate_uuid: "862f8a414df940f490cbe38227675ef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" is_pre_election: true I0918 20:38:28.708412 21211 raft_consensus.cc:2350] T 542bf88d39f1442d94d8d8e9084447f7 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862f8a414df940f490cbe38227675ef4 in term 0. I0918 20:38:28.709933 21211 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "17a83b80ba4e44bb979907ef18954b69" candidate_uuid: "862f8a414df940f490cbe38227675ef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" I0918 20:38:28.712556 21281 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "17a83b80ba4e44bb979907ef18954b69" candidate_uuid: "862f8a414df940f490cbe38227675ef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1514f228147341d5baafc932fda5d90f" I0918 20:38:28.711386 21280 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "542bf88d39f1442d94d8d8e9084447f7" candidate_uuid: "862f8a414df940f490cbe38227675ef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1514f228147341d5baafc932fda5d90f" is_pre_election: true I0918 20:38:28.713678 21211 raft_consensus.cc:2890] T 17a83b80ba4e44bb979907ef18954b69 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:28.714663 21281 raft_consensus.cc:2890] T 17a83b80ba4e44bb979907ef18954b69 P 1514f228147341d5baafc932fda5d90f [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:28.737087 21281 raft_consensus.cc:2350] T 17a83b80ba4e44bb979907ef18954b69 P 1514f228147341d5baafc932fda5d90f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862f8a414df940f490cbe38227675ef4 in term 1. I0918 20:38:28.737905 20997 catalog_manager.cc:4131] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 reported cstate change: term changed from 0 to 1, leader changed from to 862f8a414df940f490cbe38227675ef4 (127.18.222.3). New cstate: current_term: 1 leader_uuid: "862f8a414df940f490cbe38227675ef4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } health_report { overall_health: UNKNOWN } } } I0918 20:38:28.739797 21211 raft_consensus.cc:2350] T 17a83b80ba4e44bb979907ef18954b69 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862f8a414df940f490cbe38227675ef4 in term 1. I0918 20:38:28.741724 21411 tablet_bootstrap.cc:590] T 1a31b5a965a7404bbda0d2779e584d53 P 1514f228147341d5baafc932fda5d90f: No blocks or log segments found. Creating new log. I0918 20:38:28.711896 21316 leader_election.cc:310] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ff36990b0c44cf7b62f9e2df4a80087, 862f8a414df940f490cbe38227675ef4; no voters: I0918 20:38:28.744433 21418 raft_consensus.cc:2644] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Leader pre-election won for term 1 W0918 20:38:28.744588 21316 leader_election.cc:348] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1514f228147341d5baafc932fda5d90f (127.18.222.2:41977): Illegal state: must be running to vote when last-logged opid is not known I0918 20:38:28.744763 21418 raft_consensus.cc:472] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:28.745105 21418 raft_consensus.cc:2890] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:28.745458 21316 leader_election.cc:310] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ff36990b0c44cf7b62f9e2df4a80087, 862f8a414df940f490cbe38227675ef4; no voters: I0918 20:38:28.746289 21419 raft_consensus.cc:2644] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:28.746693 21419 raft_consensus.cc:667] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [term 1 LEADER]: Becoming Leader. State: Replica: 862f8a414df940f490cbe38227675ef4, State: Running, Role: LEADER I0918 20:38:28.747270 21419 consensus_queue.cc:206] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } I0918 20:38:28.751324 21418 raft_consensus.cc:494] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } I0918 20:38:28.752781 21411 tablet_bootstrap.cc:439] T 1a31b5a965a7404bbda0d2779e584d53 P 1514f228147341d5baafc932fda5d90f: No bootstrap required, opened a new log I0918 20:38:28.753090 21418 leader_election.cc:296] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [CANDIDATE]: Term 1 election: Requested vote from peers 1514f228147341d5baafc932fda5d90f (127.18.222.2:41977), 3ff36990b0c44cf7b62f9e2df4a80087 (127.18.222.1:41435) I0918 20:38:28.753388 21411 ts_tablet_manager.cc:1085] T 1a31b5a965a7404bbda0d2779e584d53 P 1514f228147341d5baafc932fda5d90f: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.005s I0918 20:38:28.754608 21211 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "542bf88d39f1442d94d8d8e9084447f7" candidate_uuid: "862f8a414df940f490cbe38227675ef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" I0918 20:38:28.755221 21211 raft_consensus.cc:2890] T 542bf88d39f1442d94d8d8e9084447f7 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:28.761023 21281 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "542bf88d39f1442d94d8d8e9084447f7" candidate_uuid: "862f8a414df940f490cbe38227675ef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1514f228147341d5baafc932fda5d90f" W0918 20:38:28.762622 21316 leader_election.cc:348] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1514f228147341d5baafc932fda5d90f (127.18.222.2:41977): Illegal state: must be running to vote when last-logged opid is not known I0918 20:38:28.766675 21211 raft_consensus.cc:2350] T 542bf88d39f1442d94d8d8e9084447f7 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862f8a414df940f490cbe38227675ef4 in term 1. I0918 20:38:28.767859 21316 leader_election.cc:310] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ff36990b0c44cf7b62f9e2df4a80087, 862f8a414df940f490cbe38227675ef4; no voters: 1514f228147341d5baafc932fda5d90f I0918 20:38:28.769134 21419 raft_consensus.cc:2644] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:28.770337 21419 raft_consensus.cc:667] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [term 1 LEADER]: Becoming Leader. State: Replica: 862f8a414df940f490cbe38227675ef4, State: Running, Role: LEADER I0918 20:38:28.770941 21419 consensus_queue.cc:206] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } I0918 20:38:28.777032 21411 raft_consensus.cc:340] T 1a31b5a965a7404bbda0d2779e584d53 P 1514f228147341d5baafc932fda5d90f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } I0918 20:38:28.777968 21411 raft_consensus.cc:366] T 1a31b5a965a7404bbda0d2779e584d53 P 1514f228147341d5baafc932fda5d90f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:28.778308 21411 raft_consensus.cc:705] T 1a31b5a965a7404bbda0d2779e584d53 P 1514f228147341d5baafc932fda5d90f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1514f228147341d5baafc932fda5d90f, State: Initialized, Role: FOLLOWER I0918 20:38:28.779171 21411 consensus_queue.cc:229] T 1a31b5a965a7404bbda0d2779e584d53 P 1514f228147341d5baafc932fda5d90f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } I0918 20:38:28.786041 20997 catalog_manager.cc:4131] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 reported cstate change: term changed from 0 to 1, leader changed from to 862f8a414df940f490cbe38227675ef4 (127.18.222.3). New cstate: current_term: 1 leader_uuid: "862f8a414df940f490cbe38227675ef4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } health_report { overall_health: HEALTHY } } } I0918 20:38:28.787623 20997 catalog_manager.cc:4131] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 reported cstate change: term changed from 0 to 1, leader changed from to 862f8a414df940f490cbe38227675ef4 (127.18.222.3). New cstate: current_term: 1 leader_uuid: "862f8a414df940f490cbe38227675ef4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } health_report { overall_health: HEALTHY } } } I0918 20:38:28.795135 21411 ts_tablet_manager.cc:1113] T 1a31b5a965a7404bbda0d2779e584d53 P 1514f228147341d5baafc932fda5d90f: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s I0918 20:38:28.795953 21411 tablet_bootstrap.cc:439] T 542bf88d39f1442d94d8d8e9084447f7 P 1514f228147341d5baafc932fda5d90f: Bootstrap starting. I0918 20:38:28.799464 21411 tablet_bootstrap.cc:590] T 542bf88d39f1442d94d8d8e9084447f7 P 1514f228147341d5baafc932fda5d90f: No blocks or log segments found. Creating new log. I0918 20:38:28.823601 21411 tablet_bootstrap.cc:439] T 542bf88d39f1442d94d8d8e9084447f7 P 1514f228147341d5baafc932fda5d90f: No bootstrap required, opened a new log I0918 20:38:28.824213 21411 ts_tablet_manager.cc:1085] T 542bf88d39f1442d94d8d8e9084447f7 P 1514f228147341d5baafc932fda5d90f: Time spent bootstrapping tablet: real 0.028s user 0.023s sys 0.003s I0918 20:38:28.826373 21411 raft_consensus.cc:340] T 542bf88d39f1442d94d8d8e9084447f7 P 1514f228147341d5baafc932fda5d90f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } I0918 20:38:28.827036 21411 raft_consensus.cc:366] T 542bf88d39f1442d94d8d8e9084447f7 P 1514f228147341d5baafc932fda5d90f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:28.827270 21411 raft_consensus.cc:705] T 542bf88d39f1442d94d8d8e9084447f7 P 1514f228147341d5baafc932fda5d90f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1514f228147341d5baafc932fda5d90f, State: Initialized, Role: FOLLOWER I0918 20:38:28.828069 21411 consensus_queue.cc:229] T 542bf88d39f1442d94d8d8e9084447f7 P 1514f228147341d5baafc932fda5d90f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } I0918 20:38:28.830390 21411 ts_tablet_manager.cc:1113] T 542bf88d39f1442d94d8d8e9084447f7 P 1514f228147341d5baafc932fda5d90f: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s I0918 20:38:29.025312 21416 raft_consensus.cc:472] T 1a31b5a965a7404bbda0d2779e584d53 P 1514f228147341d5baafc932fda5d90f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:29.025787 21416 raft_consensus.cc:494] T 1a31b5a965a7404bbda0d2779e584d53 P 1514f228147341d5baafc932fda5d90f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } I0918 20:38:29.028125 21416 leader_election.cc:296] T 1a31b5a965a7404bbda0d2779e584d53 P 1514f228147341d5baafc932fda5d90f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862f8a414df940f490cbe38227675ef4 (127.18.222.3:45615), 3ff36990b0c44cf7b62f9e2df4a80087 (127.18.222.1:41435) I0918 20:38:29.095979 21353 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "1a31b5a965a7404bbda0d2779e584d53" candidate_uuid: "1514f228147341d5baafc932fda5d90f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862f8a414df940f490cbe38227675ef4" is_pre_election: true I0918 20:38:29.096796 21353 raft_consensus.cc:2311] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 1514f228147341d5baafc932fda5d90f for term 1 because replica is either leader or believes a valid leader to be alive. I0918 20:38:29.114617 21211 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "1a31b5a965a7404bbda0d2779e584d53" candidate_uuid: "1514f228147341d5baafc932fda5d90f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" is_pre_election: true I0918 20:38:29.115449 21211 raft_consensus.cc:2280] T 1a31b5a965a7404bbda0d2779e584d53 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1514f228147341d5baafc932fda5d90f in current term 1: Already voted for candidate 862f8a414df940f490cbe38227675ef4 in this term. I0918 20:38:29.116672 21245 leader_election.cc:310] T 1a31b5a965a7404bbda0d2779e584d53 P 1514f228147341d5baafc932fda5d90f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1514f228147341d5baafc932fda5d90f; no voters: 3ff36990b0c44cf7b62f9e2df4a80087, 862f8a414df940f490cbe38227675ef4 I0918 20:38:29.117882 21416 raft_consensus.cc:2890] T 1a31b5a965a7404bbda0d2779e584d53 P 1514f228147341d5baafc932fda5d90f [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:29.130920 21416 raft_consensus.cc:2592] T 1a31b5a965a7404bbda0d2779e584d53 P 1514f228147341d5baafc932fda5d90f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0918 20:38:29.141959 21416 raft_consensus.cc:472] T 542bf88d39f1442d94d8d8e9084447f7 P 1514f228147341d5baafc932fda5d90f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:29.142525 21416 raft_consensus.cc:494] T 542bf88d39f1442d94d8d8e9084447f7 P 1514f228147341d5baafc932fda5d90f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 } } peers { permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 } } peers { permanent_uuid: "862f8a414df940f490cbe38227675ef4" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45615 } } I0918 20:38:29.146350 21211 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "542bf88d39f1442d94d8d8e9084447f7" candidate_uuid: "1514f228147341d5baafc932fda5d90f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" is_pre_election: true I0918 20:38:29.149260 21211 raft_consensus.cc:2280] T 542bf88d39f1442d94d8d8e9084447f7 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1514f228147341d5baafc932fda5d90f in current term 1: Already voted for candidate 862f8a414df940f490cbe38227675ef4 in this term. I0918 20:38:29.147233 21416 leader_election.cc:296] T 542bf88d39f1442d94d8d8e9084447f7 P 1514f228147341d5baafc932fda5d90f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ff36990b0c44cf7b62f9e2df4a80087 (127.18.222.1:41435), 862f8a414df940f490cbe38227675ef4 (127.18.222.3:45615) I0918 20:38:29.151979 21353 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "542bf88d39f1442d94d8d8e9084447f7" candidate_uuid: "1514f228147341d5baafc932fda5d90f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862f8a414df940f490cbe38227675ef4" is_pre_election: true I0918 20:38:29.152714 21353 raft_consensus.cc:2311] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 1514f228147341d5baafc932fda5d90f for term 1 because replica is either leader or believes a valid leader to be alive. I0918 20:38:29.155746 21245 leader_election.cc:310] T 542bf88d39f1442d94d8d8e9084447f7 P 1514f228147341d5baafc932fda5d90f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1514f228147341d5baafc932fda5d90f; no voters: 3ff36990b0c44cf7b62f9e2df4a80087, 862f8a414df940f490cbe38227675ef4 I0918 20:38:29.156858 21416 raft_consensus.cc:2890] T 542bf88d39f1442d94d8d8e9084447f7 P 1514f228147341d5baafc932fda5d90f [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:29.163568 21416 raft_consensus.cc:2592] T 542bf88d39f1442d94d8d8e9084447f7 P 1514f228147341d5baafc932fda5d90f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0918 20:38:29.177769 21419 consensus_queue.cc:984] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:29.187831 21419 consensus_queue.cc:984] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:29.234292 21419 consensus_queue.cc:984] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0918 20:38:29.381723 21315 outbound_call.cc:287] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.18.222.60:44745, user_credentials={real_user=slave}} blocked reactor thread for 91496.3us I0918 20:38:29.408535 21418 consensus_queue.cc:984] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:29.412608 21418 consensus_queue.cc:984] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ff36990b0c44cf7b62f9e2df4a80087" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:29.422571 21418 consensus_queue.cc:984] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1514f228147341d5baafc932fda5d90f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:29.782137 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:29.813154 21165 messenger.cc:381] Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer W0918 20:38:29.814569 21316 consensus_peers.cc:458] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 -> Peer 3ff36990b0c44cf7b62f9e2df4a80087 (127.18.222.1:41435): Couldn't send request to peer 3ff36990b0c44cf7b62f9e2df4a80087. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry. I0918 20:38:29.857496 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:29.857856 19320 tablet_replica.cc:273] T 17a83b80ba4e44bb979907ef18954b69 P 3ff36990b0c44cf7b62f9e2df4a80087: stopping tablet replica I0918 20:38:29.858238 19320 raft_consensus.cc:2147] T 17a83b80ba4e44bb979907ef18954b69 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:29.858584 19320 raft_consensus.cc:2174] T 17a83b80ba4e44bb979907ef18954b69 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:29.861053 19320 tablet_replica.cc:273] T 542bf88d39f1442d94d8d8e9084447f7 P 3ff36990b0c44cf7b62f9e2df4a80087: stopping tablet replica I0918 20:38:29.861541 19320 raft_consensus.cc:2147] T 542bf88d39f1442d94d8d8e9084447f7 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:29.861899 19320 raft_consensus.cc:2174] T 542bf88d39f1442d94d8d8e9084447f7 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:29.866441 19320 tablet_replica.cc:273] T 1a31b5a965a7404bbda0d2779e584d53 P 3ff36990b0c44cf7b62f9e2df4a80087: stopping tablet replica I0918 20:38:29.866906 19320 raft_consensus.cc:2147] T 1a31b5a965a7404bbda0d2779e584d53 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:29.867274 19320 raft_consensus.cc:2174] T 1a31b5a965a7404bbda0d2779e584d53 P 3ff36990b0c44cf7b62f9e2df4a80087 [term 1 FOLLOWER]: Raft consensus is shut down! W0918 20:38:29.905153 21316 consensus_peers.cc:458] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 -> Peer 3ff36990b0c44cf7b62f9e2df4a80087 (127.18.222.1:41435): Couldn't send request to peer 3ff36990b0c44cf7b62f9e2df4a80087. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.1:41435: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:38:30.023867 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:30.024133 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:30.024586 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:30.038290 19320 tablet_server.cc:155] TabletServer shutting down... W0918 20:38:30.064285 21316 consensus_peers.cc:458] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 -> Peer 3ff36990b0c44cf7b62f9e2df4a80087 (127.18.222.1:41435): Couldn't send request to peer 3ff36990b0c44cf7b62f9e2df4a80087. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.1:41435: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:38:30.150641 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:30.151154 19320 tablet_replica.cc:273] T 542bf88d39f1442d94d8d8e9084447f7 P 1514f228147341d5baafc932fda5d90f: stopping tablet replica I0918 20:38:30.151643 19320 raft_consensus.cc:2147] T 542bf88d39f1442d94d8d8e9084447f7 P 1514f228147341d5baafc932fda5d90f [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:30.152117 19320 raft_consensus.cc:2174] T 542bf88d39f1442d94d8d8e9084447f7 P 1514f228147341d5baafc932fda5d90f [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:30.155649 19320 tablet_replica.cc:273] T 1a31b5a965a7404bbda0d2779e584d53 P 1514f228147341d5baafc932fda5d90f: stopping tablet replica I0918 20:38:30.156179 19320 raft_consensus.cc:2147] T 1a31b5a965a7404bbda0d2779e584d53 P 1514f228147341d5baafc932fda5d90f [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:30.156750 19320 raft_consensus.cc:2174] T 1a31b5a965a7404bbda0d2779e584d53 P 1514f228147341d5baafc932fda5d90f [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:30.161793 19320 tablet_replica.cc:273] T 17a83b80ba4e44bb979907ef18954b69 P 1514f228147341d5baafc932fda5d90f: stopping tablet replica I0918 20:38:30.162257 19320 raft_consensus.cc:2147] T 17a83b80ba4e44bb979907ef18954b69 P 1514f228147341d5baafc932fda5d90f [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:30.162587 19320 raft_consensus.cc:2174] T 17a83b80ba4e44bb979907ef18954b69 P 1514f228147341d5baafc932fda5d90f [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:30.203647 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:30.204943 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:30.205242 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:30.219801 19320 tablet_server.cc:155] TabletServer shutting down... W0918 20:38:30.220366 21316 consensus_peers.cc:458] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 -> Peer 1514f228147341d5baafc932fda5d90f (127.18.222.2:41977): Couldn't send request to peer 1514f228147341d5baafc932fda5d90f. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.2:41977: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:38:30.260514 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:30.260870 19320 tablet_replica.cc:273] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4: stopping tablet replica I0918 20:38:30.261376 19320 raft_consensus.cc:2147] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:30.262230 19320 raft_consensus.cc:2174] T 542bf88d39f1442d94d8d8e9084447f7 P 862f8a414df940f490cbe38227675ef4 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:30.265044 19320 tablet_replica.cc:273] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4: stopping tablet replica I0918 20:38:30.265386 19320 raft_consensus.cc:2147] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:30.266052 19320 raft_consensus.cc:2174] T 1a31b5a965a7404bbda0d2779e584d53 P 862f8a414df940f490cbe38227675ef4 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:30.268630 19320 tablet_replica.cc:273] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4: stopping tablet replica I0918 20:38:30.268971 19320 raft_consensus.cc:2147] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:30.269731 19320 raft_consensus.cc:2174] T 17a83b80ba4e44bb979907ef18954b69 P 862f8a414df940f490cbe38227675ef4 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:30.293664 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:30.293846 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:30.294152 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:30.305446 19320 master.cc:247] Master@127.18.222.62:32943 shutting down... I0918 20:38:30.325271 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:30.326076 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:30.326354 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P fbf7f3ed055e466888130f93b574c7b5: stopping tablet replica I0918 20:38:30.346297 19320 master.cc:258] Master@127.18.222.62:32943 shutdown complete. I0918 20:38:30.357533 19320 master.cc:247] Master@127.18.222.61:40493 shutting down... I0918 20:38:30.378032 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:30.378463 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:30.378708 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 74d1e2b653404613bbdfa0e3c67634b7: stopping tablet replica I0918 20:38:30.397508 19320 master.cc:258] Master@127.18.222.61:40493 shutdown complete. I0918 20:38:30.406921 19320 master.cc:247] Master@127.18.222.60:44745 shutting down... I0918 20:38:30.426919 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:30.427376 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:30.427620 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 2362c3ddd9814f6b9cf972f09ea443fb: stopping tablet replica W0918 20:38:30.934475 21097 debug-util.cc:397] Leaking SignalData structure 0x7b080010e880 after lost signal to thread 19323 W0918 20:38:31.444231 19320 thread.cc:499] Waited for 1000ms trying to join with diag-logger (tid 21097) W0918 20:38:32.444651 19320 thread.cc:499] Waited for 2000ms trying to join with diag-logger (tid 21097) I0918 20:38:33.311777 19320 master.cc:258] Master@127.18.222.60:44745 shutdown complete. [ OK ] RemoteKsckTest.TestTableConsistency (6206 ms) [ RUN ] RemoteKsckTest.TestChecksum I0918 20:38:33.353440 19320 test_util.cc:212] Using random seed: -770725693 I0918 20:38:33.354254 19320 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.18.222.62:40939,127.18.222.61:34847,127.18.222.60:39895 I0918 20:38:33.366886 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 134092us I0918 20:38:33.367296 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:33.367446 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:33.367650 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:33.368032 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:33.369087 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/master-0-root/instance: uuid: "8eaf326bfbed476987e5379cbccca3c8" format_stamp: "Formatted at 2019-09-18 20:38:33 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:33.377729 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.008s sys 0.001s I0918 20:38:33.379199 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:33.385996 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:33.386180 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/master-0-root uuid: "8eaf326bfbed476987e5379cbccca3c8" format_stamp: "Formatted at 2019-09-18 20:38:33 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:33.386411 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:33.438516 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:40939 I0918 20:38:33.440665 19320 webserver.cc:374] Webserver started at http://127.18.222.62:44707/ using document root and password file I0918 20:38:33.443066 21533 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:33.450870 21533 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:33.463955 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 134092us I0918 20:38:33.464375 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:33.464540 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:33.464754 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:33.464880 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:33.465925 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/master-1-root/instance: uuid: "c5165bb9c7334058a7bb869966482ee5" format_stamp: "Formatted at 2019-09-18 20:38:33 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:33.469318 21533 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } has no permanent_uuid. Determining permanent_uuid... W0918 20:38:33.470988 21533 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:34847: Network error: Client connection negotiation failed: client connection to 127.18.222.61:34847: connect: Connection refused (error 111) I0918 20:38:33.476630 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.010s sys 0.003s I0918 20:38:33.478219 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:33.486295 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s I0918 20:38:33.486536 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/master-1-root uuid: "c5165bb9c7334058a7bb869966482ee5" format_stamp: "Formatted at 2019-09-18 20:38:33 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:33.486789 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:33.494365 21533 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } attempt: 1 W0918 20:38:33.496163 21533 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:34847: Network error: Client connection negotiation failed: client connection to 127.18.222.61:34847: connect: Connection refused (error 111) I0918 20:38:33.534557 21533 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } attempt: 2 W0918 20:38:33.536589 21533 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:34847: Network error: Client connection negotiation failed: client connection to 127.18.222.61:34847: connect: Connection refused (error 111) I0918 20:38:33.564491 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:34847 I0918 20:38:33.566550 19320 webserver.cc:374] Webserver started at http://127.18.222.61:33531/ using document root and password file I0918 20:38:33.568958 21590 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:33.576066 21590 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:33.589545 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 134092us I0918 20:38:33.589777 21590 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:33.589977 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:33.590268 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:33.590498 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:33.590677 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:33.592247 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/master-2-root/instance: uuid: "23a446a27d414a34bbdab5f72442acfa" format_stamp: "Formatted at 2019-09-18 20:38:33 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:33.607957 21590 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:33.608655 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.016s user 0.013s sys 0.001s W0918 20:38:33.609719 21590 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:39895: Network error: Client connection negotiation failed: client connection to 127.18.222.60:39895: connect: Connection refused (error 111) I0918 20:38:33.610232 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:33.613039 21533 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } attempt: 3 I0918 20:38:33.619748 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:38:33.620020 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/master-2-root uuid: "23a446a27d414a34bbdab5f72442acfa" format_stamp: "Formatted at 2019-09-18 20:38:33 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:33.620383 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:33.622853 21533 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } has no permanent_uuid. Determining permanent_uuid... W0918 20:38:33.624903 21533 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:39895: Network error: Client connection negotiation failed: client connection to 127.18.222.60:39895: connect: Connection refused (error 111) I0918 20:38:33.650496 21533 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } attempt: 1 W0918 20:38:33.652575 21533 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:39895: Network error: Client connection negotiation failed: client connection to 127.18.222.60:39895: connect: Connection refused (error 111) I0918 20:38:33.675123 21590 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } attempt: 1 W0918 20:38:33.677639 21590 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:39895: Network error: Client connection negotiation failed: client connection to 127.18.222.60:39895: connect: Connection refused (error 111) I0918 20:38:33.684677 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:39895 I0918 20:38:33.686492 19320 webserver.cc:374] Webserver started at http://127.18.222.60:35771/ using document root and password file I0918 20:38:33.687583 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0918 20:38:33.688655 21647 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:33.693253 21647 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:33.703780 21647 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:33.709039 21533 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } attempt: 2 I0918 20:38:33.713519 21647 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:33.727481 21533 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8: Bootstrap starting. I0918 20:38:33.730020 21647 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa: Bootstrap starting. I0918 20:38:33.730492 21533 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8: No blocks or log segments found. Creating new log. I0918 20:38:33.733187 21647 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa: No blocks or log segments found. Creating new log. I0918 20:38:33.733877 21533 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8: No bootstrap required, opened a new log I0918 20:38:33.735105 21590 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } attempt: 2 I0918 20:38:33.737365 21647 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa: No bootstrap required, opened a new log I0918 20:38:33.738698 21533 raft_consensus.cc:340] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } I0918 20:38:33.739207 21533 raft_consensus.cc:366] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:33.739419 21533 raft_consensus.cc:705] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eaf326bfbed476987e5379cbccca3c8, State: Initialized, Role: FOLLOWER I0918 20:38:33.740053 21533 consensus_queue.cc:229] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } I0918 20:38:33.741649 21647 raft_consensus.cc:340] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } I0918 20:38:33.742185 21647 raft_consensus.cc:366] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:33.742360 21647 raft_consensus.cc:705] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a446a27d414a34bbdab5f72442acfa, State: Initialized, Role: FOLLOWER I0918 20:38:33.742815 21647 consensus_queue.cc:229] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } I0918 20:38:33.743949 21533 sys_catalog.cc:340] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } } I0918 20:38:33.744820 21533 sys_catalog.cc:343] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:33.744359 21653 sys_catalog.cc:340] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } } I0918 20:38:33.745429 21653 sys_catalog.cc:343] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:33.745252 21647 sys_catalog.cc:340] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } } I0918 20:38:33.746047 21647 sys_catalog.cc:343] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:33.746248 21533 sys_catalog.cc:428] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:38:33.746764 21654 sys_catalog.cc:340] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } } I0918 20:38:33.747771 21654 sys_catalog.cc:343] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:33.747838 21647 sys_catalog.cc:428] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa [sys.catalog]: configured and running, proceeding with master startup. I0918 20:38:33.748540 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 W0918 20:38:33.751750 21655 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:38:33.752046 21655 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:33.753304 21590 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5: Bootstrap starting. I0918 20:38:33.756106 21590 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5: No blocks or log segments found. Creating new log. I0918 20:38:33.759492 21590 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5: No bootstrap required, opened a new log W0918 20:38:33.761010 21656 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:38:33.761297 21656 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:33.761632 21590 raft_consensus.cc:340] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } I0918 20:38:33.762118 21590 raft_consensus.cc:366] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:33.762280 21590 raft_consensus.cc:705] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5165bb9c7334058a7bb869966482ee5, State: Initialized, Role: FOLLOWER I0918 20:38:33.762713 21590 consensus_queue.cc:229] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } I0918 20:38:33.764161 21590 sys_catalog.cc:340] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } } I0918 20:38:33.764654 21590 sys_catalog.cc:343] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:33.764380 21658 sys_catalog.cc:340] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } } I0918 20:38:33.765012 21658 sys_catalog.cc:343] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:33.765554 21590 sys_catalog.cc:428] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:38:33.766477 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 W0918 20:38:33.769287 21659 catalog_manager.cc:1155] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:38:33.769587 21659 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:33.783373 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 134092us I0918 20:38:33.783758 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:33.783905 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:33.784097 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:33.784224 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:33.785212 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/ts-0-root/instance: uuid: "5a30c225d1e543d69c2601bde2c33afc" format_stamp: "Formatted at 2019-09-18 20:38:33 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:33.795517 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.011s sys 0.000s I0918 20:38:33.797176 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:33.805402 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:38:33.805588 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/ts-0-root uuid: "5a30c225d1e543d69c2601bde2c33afc" format_stamp: "Formatted at 2019-09-18 20:38:33 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:33.805829 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:33.843760 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:33.843937 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:33.884526 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:39639 I0918 20:38:33.886369 19320 webserver.cc:374] Webserver started at http://127.18.222.1:41661/ using document root and password file I0918 20:38:33.919402 21724 heartbeater.cc:322] Connected to a master server at 127.18.222.60:39895 I0918 20:38:33.919462 21726 heartbeater.cc:322] Connected to a master server at 127.18.222.62:40939 I0918 20:38:33.919651 21725 heartbeater.cc:322] Connected to a master server at 127.18.222.61:34847 I0918 20:38:33.919816 21724 heartbeater.cc:406] Registering TS with master... I0918 20:38:33.920022 21726 heartbeater.cc:406] Registering TS with master... I0918 20:38:33.920089 21725 heartbeater.cc:406] Registering TS with master... I0918 20:38:33.920565 21724 heartbeater.cc:455] Master 127.18.222.60:39895 requested a full tablet report, sending... I0918 20:38:33.920799 21726 heartbeater.cc:455] Master 127.18.222.62:40939 requested a full tablet report, sending... I0918 20:38:33.920892 21725 heartbeater.cc:455] Master 127.18.222.61:34847 requested a full tablet report, sending... I0918 20:38:33.923367 21500 ts_manager.cc:186] Registered new tserver with Master: 5a30c225d1e543d69c2601bde2c33afc (127.18.222.1:39639) I0918 20:38:33.923964 21557 ts_manager.cc:186] Registered new tserver with Master: 5a30c225d1e543d69c2601bde2c33afc (127.18.222.1:39639) I0918 20:38:33.923337 21614 ts_manager.cc:186] Registered new tserver with Master: 5a30c225d1e543d69c2601bde2c33afc (127.18.222.1:39639) I0918 20:38:33.929330 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 134092us I0918 20:38:33.929683 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:33.929831 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:33.930013 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:33.930127 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:33.931061 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/ts-1-root/instance: uuid: "a2535df90af94322a63997ab490e9f7b" format_stamp: "Formatted at 2019-09-18 20:38:33 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:33.943859 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.012s user 0.013s sys 0.000s I0918 20:38:33.945441 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:33.956312 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:33.956523 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/ts-1-root uuid: "a2535df90af94322a63997ab490e9f7b" format_stamp: "Formatted at 2019-09-18 20:38:33 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:33.956748 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:33.984426 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:33.984622 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:34.046329 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.2:43981 I0918 20:38:34.048375 19320 webserver.cc:374] Webserver started at http://127.18.222.2:34287/ using document root and password file I0918 20:38:34.092869 21795 heartbeater.cc:322] Connected to a master server at 127.18.222.60:39895 I0918 20:38:34.093252 21795 heartbeater.cc:406] Registering TS with master... I0918 20:38:34.093861 21795 heartbeater.cc:455] Master 127.18.222.60:39895 requested a full tablet report, sending... I0918 20:38:34.095893 21796 heartbeater.cc:322] Connected to a master server at 127.18.222.61:34847 I0918 20:38:34.096199 21796 heartbeater.cc:406] Registering TS with master... I0918 20:38:34.096582 21614 ts_manager.cc:186] Registered new tserver with Master: a2535df90af94322a63997ab490e9f7b (127.18.222.2:43981) I0918 20:38:34.096817 21796 heartbeater.cc:455] Master 127.18.222.61:34847 requested a full tablet report, sending... I0918 20:38:34.098997 21557 ts_manager.cc:186] Registered new tserver with Master: a2535df90af94322a63997ab490e9f7b (127.18.222.2:43981) I0918 20:38:34.104035 21653 raft_consensus.cc:472] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:34.104408 21797 heartbeater.cc:322] Connected to a master server at 127.18.222.62:40939 I0918 20:38:34.104836 21797 heartbeater.cc:406] Registering TS with master... I0918 20:38:34.104502 21653 raft_consensus.cc:494] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } I0918 20:38:34.105373 21797 heartbeater.cc:455] Master 127.18.222.62:40939 requested a full tablet report, sending... I0918 20:38:34.106132 21653 leader_election.cc:296] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5165bb9c7334058a7bb869966482ee5 (127.18.222.61:34847), 23a446a27d414a34bbdab5f72442acfa (127.18.222.60:39895) I0918 20:38:34.107086 21567 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8eaf326bfbed476987e5379cbccca3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5165bb9c7334058a7bb869966482ee5" is_pre_election: true I0918 20:38:34.107762 21500 ts_manager.cc:186] Registered new tserver with Master: a2535df90af94322a63997ab490e9f7b (127.18.222.2:43981) I0918 20:38:34.107782 21624 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8eaf326bfbed476987e5379cbccca3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a446a27d414a34bbdab5f72442acfa" is_pre_election: true I0918 20:38:34.108260 21567 raft_consensus.cc:2350] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eaf326bfbed476987e5379cbccca3c8 in term 0. I0918 20:38:34.108642 21624 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eaf326bfbed476987e5379cbccca3c8 in term 0. I0918 20:38:34.109623 21484 leader_election.cc:310] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8eaf326bfbed476987e5379cbccca3c8, c5165bb9c7334058a7bb869966482ee5; no voters: I0918 20:38:34.110460 21653 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:34.110745 21653 raft_consensus.cc:472] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:34.111017 21653 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:34.113826 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 134592us I0918 20:38:34.114305 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:34.114521 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:34.114807 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:34.114996 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:34.116300 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/ts-2-root/instance: uuid: "772cb6045c9d4a7785077cdfe1b5312a" format_stamp: "Formatted at 2019-09-18 20:38:34 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:34.117653 21653 raft_consensus.cc:494] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } I0918 20:38:34.119848 21567 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8eaf326bfbed476987e5379cbccca3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5165bb9c7334058a7bb869966482ee5" I0918 20:38:34.120796 21567 raft_consensus.cc:2890] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:34.121385 21624 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8eaf326bfbed476987e5379cbccca3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a446a27d414a34bbdab5f72442acfa" I0918 20:38:34.121798 21624 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:34.126977 21624 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eaf326bfbed476987e5379cbccca3c8 in term 1. I0918 20:38:34.128130 21481 leader_election.cc:310] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a446a27d414a34bbdab5f72442acfa, 8eaf326bfbed476987e5379cbccca3c8; no voters: I0918 20:38:34.130282 21567 raft_consensus.cc:2350] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eaf326bfbed476987e5379cbccca3c8 in term 1. I0918 20:38:34.131031 21653 leader_election.cc:296] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [CANDIDATE]: Term 1 election: Requested vote from peers c5165bb9c7334058a7bb869966482ee5 (127.18.222.61:34847), 23a446a27d414a34bbdab5f72442acfa (127.18.222.60:39895) I0918 20:38:34.131522 21653 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:34.131916 21653 raft_consensus.cc:667] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [term 1 LEADER]: Becoming Leader. State: Replica: 8eaf326bfbed476987e5379cbccca3c8, State: Running, Role: LEADER I0918 20:38:34.132510 21653 consensus_queue.cc:206] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } I0918 20:38:34.136425 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.020s user 0.010s sys 0.005s I0918 20:38:34.137578 21802 sys_catalog.cc:340] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eaf326bfbed476987e5379cbccca3c8. Latest consensus state: current_term: 1 leader_uuid: "8eaf326bfbed476987e5379cbccca3c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } } I0918 20:38:34.138222 21802 sys_catalog.cc:343] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [sys.catalog]: This master's current role is: LEADER I0918 20:38:34.138783 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:34.139915 21805 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:38:34.143529 21805 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:38:34.153156 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I0918 20:38:34.153487 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/ts-2-root uuid: "772cb6045c9d4a7785077cdfe1b5312a" format_stamp: "Formatted at 2019-09-18 20:38:34 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:34.153811 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1568839089958813-19320/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:34.204696 21624 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa [term 1 FOLLOWER]: Refusing update from remote peer 8eaf326bfbed476987e5379cbccca3c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:34.206287 21802 consensus_queue.cc:984] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:34.209375 21567 raft_consensus.cc:1184] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5 [term 1 FOLLOWER]: Refusing update from remote peer 8eaf326bfbed476987e5379cbccca3c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:34.210741 21802 consensus_queue.cc:984] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:34.228128 21654 sys_catalog.cc:340] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eaf326bfbed476987e5379cbccca3c8. Latest consensus state: current_term: 1 leader_uuid: "8eaf326bfbed476987e5379cbccca3c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } } I0918 20:38:34.229005 21654 sys_catalog.cc:343] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:34.232465 21658 sys_catalog.cc:340] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eaf326bfbed476987e5379cbccca3c8. Latest consensus state: current_term: 1 leader_uuid: "8eaf326bfbed476987e5379cbccca3c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } } I0918 20:38:34.233618 21658 sys_catalog.cc:343] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:34.246744 21802 sys_catalog.cc:340] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8eaf326bfbed476987e5379cbccca3c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } } I0918 20:38:34.249912 21802 sys_catalog.cc:343] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [sys.catalog]: This master's current role is: LEADER I0918 20:38:34.258038 21802 sys_catalog.cc:340] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8eaf326bfbed476987e5379cbccca3c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } } I0918 20:38:34.261871 21802 sys_catalog.cc:343] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [sys.catalog]: This master's current role is: LEADER I0918 20:38:34.268657 21654 sys_catalog.cc:340] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8eaf326bfbed476987e5379cbccca3c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } } I0918 20:38:34.269659 21654 sys_catalog.cc:343] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:34.282793 21658 sys_catalog.cc:340] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8eaf326bfbed476987e5379cbccca3c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaf326bfbed476987e5379cbccca3c8" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40939 } } peers { permanent_uuid: "c5165bb9c7334058a7bb869966482ee5" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 34847 } } peers { permanent_uuid: "23a446a27d414a34bbdab5f72442acfa" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39895 } } } I0918 20:38:34.283385 21658 sys_catalog.cc:343] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:34.284853 21805 catalog_manager.cc:981] Generated new certificate authority record I0918 20:38:34.285627 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:34.285934 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:34.287043 21805 catalog_manager.cc:1106] Loading token signing keys... I0918 20:38:34.326452 21805 catalog_manager.cc:4446] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8: Generated new TSK 0 I0918 20:38:34.328291 21805 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:38:34.342406 21500 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50968 I0918 20:38:34.345592 21499 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50962 I0918 20:38:34.357228 21797 heartbeater.cc:447] Master 127.18.222.62:40939 was elected leader, sending a full tablet report... I0918 20:38:34.359546 21726 heartbeater.cc:447] Master 127.18.222.62:40939 was elected leader, sending a full tablet report... I0918 20:38:34.379557 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.3:46835 I0918 20:38:34.381665 19320 webserver.cc:374] Webserver started at http://127.18.222.3:34755/ using document root and password file I0918 20:38:34.414861 21877 heartbeater.cc:322] Connected to a master server at 127.18.222.60:39895 I0918 20:38:34.415191 21877 heartbeater.cc:406] Registering TS with master... I0918 20:38:34.416162 21877 heartbeater.cc:455] Master 127.18.222.60:39895 requested a full tablet report, sending... I0918 20:38:34.418968 21614 ts_manager.cc:186] Registered new tserver with Master: 772cb6045c9d4a7785077cdfe1b5312a (127.18.222.3:46835) I0918 20:38:34.429771 21878 heartbeater.cc:322] Connected to a master server at 127.18.222.61:34847 I0918 20:38:34.430130 21878 heartbeater.cc:406] Registering TS with master... I0918 20:38:34.430549 21878 heartbeater.cc:455] Master 127.18.222.61:34847 requested a full tablet report, sending... I0918 20:38:34.431602 21879 heartbeater.cc:322] Connected to a master server at 127.18.222.62:40939 I0918 20:38:34.432044 21879 heartbeater.cc:406] Registering TS with master... I0918 20:38:34.432598 21879 heartbeater.cc:455] Master 127.18.222.62:40939 requested a full tablet report, sending... I0918 20:38:34.433141 21557 ts_manager.cc:186] Registered new tserver with Master: 772cb6045c9d4a7785077cdfe1b5312a (127.18.222.3:46835) I0918 20:38:34.435338 21499 ts_manager.cc:186] Registered new tserver with Master: 772cb6045c9d4a7785077cdfe1b5312a (127.18.222.3:46835) I0918 20:38:34.437502 19320 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.038020685s I0918 20:38:34.437752 21499 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50974 I0918 20:38:34.453320 21879 heartbeater.cc:447] Master 127.18.222.62:40939 was elected leader, sending a full tablet report... I0918 20:38:34.492907 21499 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50978: name: "ksck-test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\n\000\000\000\004\001\024\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:38:34.495404 21499 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:38:34.628547 21843 tablet_service.cc:1022] Processing CreateTablet for tablet 5effc580c4394916a52d24cfe1b086f2 (table=ksck-test-table [id=fed44c2d57654b06b0045ee9ebb8640b]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:34.629663 21843 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 5effc580c4394916a52d24cfe1b086f2. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:34.631822 21762 tablet_service.cc:1022] Processing CreateTablet for tablet 56087a5c23f74746b0a391195509fb7a (table=ksck-test-table [id=fed44c2d57654b06b0045ee9ebb8640b]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:34.632645 21762 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 56087a5c23f74746b0a391195509fb7a. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:34.632472 21842 tablet_service.cc:1022] Processing CreateTablet for tablet f851d4a38fc246a1ad249ee5a6eee631 (table=ksck-test-table [id=fed44c2d57654b06b0045ee9ebb8640b]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:34.633451 21842 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f851d4a38fc246a1ad249ee5a6eee631. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:34.635350 21844 tablet_service.cc:1022] Processing CreateTablet for tablet 56087a5c23f74746b0a391195509fb7a (table=ksck-test-table [id=fed44c2d57654b06b0045ee9ebb8640b]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:34.636517 21844 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 56087a5c23f74746b0a391195509fb7a. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:34.642902 21761 tablet_service.cc:1022] Processing CreateTablet for tablet 5effc580c4394916a52d24cfe1b086f2 (table=ksck-test-table [id=fed44c2d57654b06b0045ee9ebb8640b]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:34.643795 21761 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 5effc580c4394916a52d24cfe1b086f2. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:34.655206 21760 tablet_service.cc:1022] Processing CreateTablet for tablet f851d4a38fc246a1ad249ee5a6eee631 (table=ksck-test-table [id=fed44c2d57654b06b0045ee9ebb8640b]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:34.657073 21760 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f851d4a38fc246a1ad249ee5a6eee631. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:34.688194 21691 tablet_service.cc:1022] Processing CreateTablet for tablet 56087a5c23f74746b0a391195509fb7a (table=ksck-test-table [id=fed44c2d57654b06b0045ee9ebb8640b]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:34.688531 21900 tablet_bootstrap.cc:439] T 5effc580c4394916a52d24cfe1b086f2 P a2535df90af94322a63997ab490e9f7b: Bootstrap starting. I0918 20:38:34.689355 21691 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 56087a5c23f74746b0a391195509fb7a. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:34.697223 21690 tablet_service.cc:1022] Processing CreateTablet for tablet 5effc580c4394916a52d24cfe1b086f2 (table=ksck-test-table [id=fed44c2d57654b06b0045ee9ebb8640b]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:34.698608 21690 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 5effc580c4394916a52d24cfe1b086f2. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:34.705926 21689 tablet_service.cc:1022] Processing CreateTablet for tablet f851d4a38fc246a1ad249ee5a6eee631 (table=ksck-test-table [id=fed44c2d57654b06b0045ee9ebb8640b]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:34.707093 21689 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f851d4a38fc246a1ad249ee5a6eee631. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:34.721796 21900 tablet_bootstrap.cc:590] T 5effc580c4394916a52d24cfe1b086f2 P a2535df90af94322a63997ab490e9f7b: No blocks or log segments found. Creating new log. I0918 20:38:34.735383 21902 tablet_bootstrap.cc:439] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc: Bootstrap starting. I0918 20:38:34.752673 21901 tablet_bootstrap.cc:439] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a: Bootstrap starting. I0918 20:38:34.756639 21901 tablet_bootstrap.cc:590] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a: No blocks or log segments found. Creating new log. I0918 20:38:34.758036 21900 tablet_bootstrap.cc:439] T 5effc580c4394916a52d24cfe1b086f2 P a2535df90af94322a63997ab490e9f7b: No bootstrap required, opened a new log I0918 20:38:34.758673 21900 ts_tablet_manager.cc:1085] T 5effc580c4394916a52d24cfe1b086f2 P a2535df90af94322a63997ab490e9f7b: Time spent bootstrapping tablet: real 0.070s user 0.013s sys 0.014s I0918 20:38:34.759801 21902 tablet_bootstrap.cc:590] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc: No blocks or log segments found. Creating new log. I0918 20:38:34.769085 21656 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa: acquiring CA information for follower catalog manager: success I0918 20:38:34.771988 21900 raft_consensus.cc:340] T 5effc580c4394916a52d24cfe1b086f2 P a2535df90af94322a63997ab490e9f7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } I0918 20:38:34.773386 21900 raft_consensus.cc:366] T 5effc580c4394916a52d24cfe1b086f2 P a2535df90af94322a63997ab490e9f7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:34.773723 21900 raft_consensus.cc:705] T 5effc580c4394916a52d24cfe1b086f2 P a2535df90af94322a63997ab490e9f7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2535df90af94322a63997ab490e9f7b, State: Initialized, Role: FOLLOWER I0918 20:38:34.774695 21900 consensus_queue.cc:229] T 5effc580c4394916a52d24cfe1b086f2 P a2535df90af94322a63997ab490e9f7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } I0918 20:38:34.795096 21659 catalog_manager.cc:1153] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5: acquiring CA information for follower catalog manager: success I0918 20:38:34.809361 21901 tablet_bootstrap.cc:439] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a: No bootstrap required, opened a new log I0918 20:38:34.809957 21901 ts_tablet_manager.cc:1085] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a: Time spent bootstrapping tablet: real 0.058s user 0.021s sys 0.011s I0918 20:38:34.810827 21659 catalog_manager.cc:1181] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:38:34.812213 21901 raft_consensus.cc:340] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } I0918 20:38:34.812882 21901 raft_consensus.cc:366] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:34.813091 21901 raft_consensus.cc:705] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 772cb6045c9d4a7785077cdfe1b5312a, State: Initialized, Role: FOLLOWER I0918 20:38:34.813573 21901 consensus_queue.cc:229] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } I0918 20:38:34.830180 21900 ts_tablet_manager.cc:1113] T 5effc580c4394916a52d24cfe1b086f2 P a2535df90af94322a63997ab490e9f7b: Time spent starting tablet: real 0.071s user 0.007s sys 0.000s I0918 20:38:34.831027 21900 tablet_bootstrap.cc:439] T 56087a5c23f74746b0a391195509fb7a P a2535df90af94322a63997ab490e9f7b: Bootstrap starting. I0918 20:38:34.812675 21902 tablet_bootstrap.cc:439] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc: No bootstrap required, opened a new log I0918 20:38:34.834489 21902 ts_tablet_manager.cc:1085] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc: Time spent bootstrapping tablet: real 0.100s user 0.016s sys 0.025s I0918 20:38:34.834729 21900 tablet_bootstrap.cc:590] T 56087a5c23f74746b0a391195509fb7a P a2535df90af94322a63997ab490e9f7b: No blocks or log segments found. Creating new log. I0918 20:38:34.835121 21656 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:38:34.836710 21902 raft_consensus.cc:340] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } I0918 20:38:34.837960 21902 raft_consensus.cc:366] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:34.838196 21902 raft_consensus.cc:705] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a30c225d1e543d69c2601bde2c33afc, State: Initialized, Role: FOLLOWER I0918 20:38:34.839118 21901 ts_tablet_manager.cc:1113] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a: Time spent starting tablet: real 0.029s user 0.005s sys 0.004s I0918 20:38:34.839073 21902 consensus_queue.cc:229] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } I0918 20:38:34.840184 21901 tablet_bootstrap.cc:439] T f851d4a38fc246a1ad249ee5a6eee631 P 772cb6045c9d4a7785077cdfe1b5312a: Bootstrap starting. I0918 20:38:34.844856 21901 tablet_bootstrap.cc:590] T f851d4a38fc246a1ad249ee5a6eee631 P 772cb6045c9d4a7785077cdfe1b5312a: No blocks or log segments found. Creating new log. I0918 20:38:34.852221 21900 tablet_bootstrap.cc:439] T 56087a5c23f74746b0a391195509fb7a P a2535df90af94322a63997ab490e9f7b: No bootstrap required, opened a new log I0918 20:38:34.852732 21900 ts_tablet_manager.cc:1085] T 56087a5c23f74746b0a391195509fb7a P a2535df90af94322a63997ab490e9f7b: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s I0918 20:38:34.854964 21900 raft_consensus.cc:340] T 56087a5c23f74746b0a391195509fb7a P a2535df90af94322a63997ab490e9f7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } I0918 20:38:34.855584 21900 raft_consensus.cc:366] T 56087a5c23f74746b0a391195509fb7a P a2535df90af94322a63997ab490e9f7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:34.855973 21900 raft_consensus.cc:705] T 56087a5c23f74746b0a391195509fb7a P a2535df90af94322a63997ab490e9f7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2535df90af94322a63997ab490e9f7b, State: Initialized, Role: FOLLOWER I0918 20:38:34.857077 21900 consensus_queue.cc:229] T 56087a5c23f74746b0a391195509fb7a P a2535df90af94322a63997ab490e9f7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } I0918 20:38:34.874294 21902 ts_tablet_manager.cc:1113] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc: Time spent starting tablet: real 0.040s user 0.004s sys 0.003s I0918 20:38:34.875161 21902 tablet_bootstrap.cc:439] T 5effc580c4394916a52d24cfe1b086f2 P 5a30c225d1e543d69c2601bde2c33afc: Bootstrap starting. I0918 20:38:34.880143 21902 tablet_bootstrap.cc:590] T 5effc580c4394916a52d24cfe1b086f2 P 5a30c225d1e543d69c2601bde2c33afc: No blocks or log segments found. Creating new log. I0918 20:38:34.883481 21901 tablet_bootstrap.cc:439] T f851d4a38fc246a1ad249ee5a6eee631 P 772cb6045c9d4a7785077cdfe1b5312a: No bootstrap required, opened a new log I0918 20:38:34.885051 21901 ts_tablet_manager.cc:1085] T f851d4a38fc246a1ad249ee5a6eee631 P 772cb6045c9d4a7785077cdfe1b5312a: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.020s I0918 20:38:34.884371 21902 tablet_bootstrap.cc:439] T 5effc580c4394916a52d24cfe1b086f2 P 5a30c225d1e543d69c2601bde2c33afc: No bootstrap required, opened a new log I0918 20:38:34.887832 21902 ts_tablet_manager.cc:1085] T 5effc580c4394916a52d24cfe1b086f2 P 5a30c225d1e543d69c2601bde2c33afc: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s I0918 20:38:34.890221 21902 raft_consensus.cc:340] T 5effc580c4394916a52d24cfe1b086f2 P 5a30c225d1e543d69c2601bde2c33afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } I0918 20:38:34.890985 21902 raft_consensus.cc:366] T 5effc580c4394916a52d24cfe1b086f2 P 5a30c225d1e543d69c2601bde2c33afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:34.891234 21902 raft_consensus.cc:705] T 5effc580c4394916a52d24cfe1b086f2 P 5a30c225d1e543d69c2601bde2c33afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a30c225d1e543d69c2601bde2c33afc, State: Initialized, Role: FOLLOWER I0918 20:38:34.884454 21900 ts_tablet_manager.cc:1113] T 56087a5c23f74746b0a391195509fb7a P a2535df90af94322a63997ab490e9f7b: Time spent starting tablet: real 0.031s user 0.005s sys 0.000s I0918 20:38:34.892647 21900 tablet_bootstrap.cc:439] T f851d4a38fc246a1ad249ee5a6eee631 P a2535df90af94322a63997ab490e9f7b: Bootstrap starting. I0918 20:38:34.894410 21906 raft_consensus.cc:472] T 5effc580c4394916a52d24cfe1b086f2 P a2535df90af94322a63997ab490e9f7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:34.894925 21906 raft_consensus.cc:494] T 5effc580c4394916a52d24cfe1b086f2 P a2535df90af94322a63997ab490e9f7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } I0918 20:38:34.895524 21907 raft_consensus.cc:472] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:34.896051 21907 raft_consensus.cc:494] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } I0918 20:38:34.896952 21906 leader_election.cc:296] T 5effc580c4394916a52d24cfe1b086f2 P a2535df90af94322a63997ab490e9f7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a30c225d1e543d69c2601bde2c33afc (127.18.222.1:39639), 772cb6045c9d4a7785077cdfe1b5312a (127.18.222.3:46835) I0918 20:38:34.897858 21907 leader_election.cc:296] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a30c225d1e543d69c2601bde2c33afc (127.18.222.1:39639), a2535df90af94322a63997ab490e9f7b (127.18.222.2:43981) I0918 20:38:34.909505 21900 tablet_bootstrap.cc:590] T f851d4a38fc246a1ad249ee5a6eee631 P a2535df90af94322a63997ab490e9f7b: No blocks or log segments found. Creating new log. I0918 20:38:34.924196 21902 consensus_queue.cc:229] T 5effc580c4394916a52d24cfe1b086f2 P 5a30c225d1e543d69c2601bde2c33afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } I0918 20:38:34.887373 21901 raft_consensus.cc:340] T f851d4a38fc246a1ad249ee5a6eee631 P 772cb6045c9d4a7785077cdfe1b5312a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } I0918 20:38:34.926030 21901 raft_consensus.cc:366] T f851d4a38fc246a1ad249ee5a6eee631 P 772cb6045c9d4a7785077cdfe1b5312a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:34.926266 21901 raft_consensus.cc:705] T f851d4a38fc246a1ad249ee5a6eee631 P 772cb6045c9d4a7785077cdfe1b5312a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 772cb6045c9d4a7785077cdfe1b5312a, State: Initialized, Role: FOLLOWER I0918 20:38:34.926820 21901 consensus_queue.cc:229] T f851d4a38fc246a1ad249ee5a6eee631 P 772cb6045c9d4a7785077cdfe1b5312a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } I0918 20:38:34.931201 21701 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5effc580c4394916a52d24cfe1b086f2" candidate_uuid: "772cb6045c9d4a7785077cdfe1b5312a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a30c225d1e543d69c2601bde2c33afc" is_pre_election: true I0918 20:38:34.932135 21701 raft_consensus.cc:2350] T 5effc580c4394916a52d24cfe1b086f2 P 5a30c225d1e543d69c2601bde2c33afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 772cb6045c9d4a7785077cdfe1b5312a in term 0. I0918 20:38:34.933696 21809 leader_election.cc:310] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a30c225d1e543d69c2601bde2c33afc, 772cb6045c9d4a7785077cdfe1b5312a; no voters: I0918 20:38:34.935216 21907 raft_consensus.cc:2644] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:34.935591 21907 raft_consensus.cc:472] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:34.936017 21907 raft_consensus.cc:2890] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:34.949388 21772 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5effc580c4394916a52d24cfe1b086f2" candidate_uuid: "772cb6045c9d4a7785077cdfe1b5312a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2535df90af94322a63997ab490e9f7b" is_pre_election: true I0918 20:38:34.950166 21772 raft_consensus.cc:2350] T 5effc580c4394916a52d24cfe1b086f2 P a2535df90af94322a63997ab490e9f7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 772cb6045c9d4a7785077cdfe1b5312a in term 0. I0918 20:38:34.955854 21909 raft_consensus.cc:472] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:34.956349 21909 raft_consensus.cc:494] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } I0918 20:38:34.957950 21902 ts_tablet_manager.cc:1113] T 5effc580c4394916a52d24cfe1b086f2 P 5a30c225d1e543d69c2601bde2c33afc: Time spent starting tablet: real 0.070s user 0.003s sys 0.003s I0918 20:38:34.958293 21909 leader_election.cc:296] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2535df90af94322a63997ab490e9f7b (127.18.222.2:43981), 772cb6045c9d4a7785077cdfe1b5312a (127.18.222.3:46835) I0918 20:38:34.958794 21902 tablet_bootstrap.cc:439] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc: Bootstrap starting. I0918 20:38:34.955332 21907 raft_consensus.cc:494] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } I0918 20:38:34.978065 21901 ts_tablet_manager.cc:1113] T f851d4a38fc246a1ad249ee5a6eee631 P 772cb6045c9d4a7785077cdfe1b5312a: Time spent starting tablet: real 0.093s user 0.003s sys 0.003s I0918 20:38:34.997848 21772 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5effc580c4394916a52d24cfe1b086f2" candidate_uuid: "772cb6045c9d4a7785077cdfe1b5312a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2535df90af94322a63997ab490e9f7b" I0918 20:38:34.998648 21772 raft_consensus.cc:2890] T 5effc580c4394916a52d24cfe1b086f2 P a2535df90af94322a63997ab490e9f7b [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:34.994096 21907 leader_election.cc:296] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [CANDIDATE]: Term 1 election: Requested vote from peers 5a30c225d1e543d69c2601bde2c33afc (127.18.222.1:39639), a2535df90af94322a63997ab490e9f7b (127.18.222.2:43981) I0918 20:38:34.995602 21901 tablet_bootstrap.cc:439] T 56087a5c23f74746b0a391195509fb7a P 772cb6045c9d4a7785077cdfe1b5312a: Bootstrap starting. I0918 20:38:35.007239 21772 raft_consensus.cc:2350] T 5effc580c4394916a52d24cfe1b086f2 P a2535df90af94322a63997ab490e9f7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 772cb6045c9d4a7785077cdfe1b5312a in term 1. I0918 20:38:35.023296 21809 leader_election.cc:310] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 772cb6045c9d4a7785077cdfe1b5312a, a2535df90af94322a63997ab490e9f7b; no voters: I0918 20:38:35.024547 21907 raft_consensus.cc:2644] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:35.024801 21900 tablet_bootstrap.cc:439] T f851d4a38fc246a1ad249ee5a6eee631 P a2535df90af94322a63997ab490e9f7b: No bootstrap required, opened a new log I0918 20:38:35.026520 21854 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5effc580c4394916a52d24cfe1b086f2" candidate_uuid: "a2535df90af94322a63997ab490e9f7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "772cb6045c9d4a7785077cdfe1b5312a" is_pre_election: true I0918 20:38:35.033509 21701 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5effc580c4394916a52d24cfe1b086f2" candidate_uuid: "772cb6045c9d4a7785077cdfe1b5312a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a30c225d1e543d69c2601bde2c33afc" I0918 20:38:35.037050 21902 tablet_bootstrap.cc:590] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc: No blocks or log segments found. Creating new log. I0918 20:38:35.034562 21900 ts_tablet_manager.cc:1085] T f851d4a38fc246a1ad249ee5a6eee631 P a2535df90af94322a63997ab490e9f7b: Time spent bootstrapping tablet: real 0.142s user 0.018s sys 0.004s I0918 20:38:35.028122 21901 tablet_bootstrap.cc:590] T 56087a5c23f74746b0a391195509fb7a P 772cb6045c9d4a7785077cdfe1b5312a: No blocks or log segments found. Creating new log. I0918 20:38:35.038995 21701 raft_consensus.cc:2890] T 5effc580c4394916a52d24cfe1b086f2 P 5a30c225d1e543d69c2601bde2c33afc [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:35.049557 21907 raft_consensus.cc:667] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [term 1 LEADER]: Becoming Leader. State: Replica: 772cb6045c9d4a7785077cdfe1b5312a, State: Running, Role: LEADER I0918 20:38:35.055092 21907 consensus_queue.cc:206] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } I0918 20:38:35.057231 21900 raft_consensus.cc:340] T f851d4a38fc246a1ad249ee5a6eee631 P a2535df90af94322a63997ab490e9f7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } I0918 20:38:35.058066 21900 raft_consensus.cc:366] T f851d4a38fc246a1ad249ee5a6eee631 P a2535df90af94322a63997ab490e9f7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:35.058317 21900 raft_consensus.cc:705] T f851d4a38fc246a1ad249ee5a6eee631 P a2535df90af94322a63997ab490e9f7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2535df90af94322a63997ab490e9f7b, State: Initialized, Role: FOLLOWER I0918 20:38:35.058894 21900 consensus_queue.cc:229] T f851d4a38fc246a1ad249ee5a6eee631 P a2535df90af94322a63997ab490e9f7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } I0918 20:38:35.067925 21700 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5effc580c4394916a52d24cfe1b086f2" candidate_uuid: "a2535df90af94322a63997ab490e9f7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a30c225d1e543d69c2601bde2c33afc" is_pre_election: true I0918 20:38:35.063979 21701 raft_consensus.cc:2350] T 5effc580c4394916a52d24cfe1b086f2 P 5a30c225d1e543d69c2601bde2c33afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 772cb6045c9d4a7785077cdfe1b5312a in term 1. I0918 20:38:35.071179 21700 raft_consensus.cc:2325] T 5effc580c4394916a52d24cfe1b086f2 P 5a30c225d1e543d69c2601bde2c33afc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2535df90af94322a63997ab490e9f7b for term 1 because replica is already servicing an update from a current leader or another vote. I0918 20:38:35.082406 21900 ts_tablet_manager.cc:1113] T f851d4a38fc246a1ad249ee5a6eee631 P a2535df90af94322a63997ab490e9f7b: Time spent starting tablet: real 0.043s user 0.006s sys 0.000s I0918 20:38:35.083918 21772 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f851d4a38fc246a1ad249ee5a6eee631" candidate_uuid: "5a30c225d1e543d69c2601bde2c33afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2535df90af94322a63997ab490e9f7b" is_pre_election: true I0918 20:38:35.084579 21772 raft_consensus.cc:2350] T f851d4a38fc246a1ad249ee5a6eee631 P a2535df90af94322a63997ab490e9f7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a30c225d1e543d69c2601bde2c33afc in term 0. I0918 20:38:35.085809 21664 leader_election.cc:310] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a30c225d1e543d69c2601bde2c33afc, a2535df90af94322a63997ab490e9f7b; no voters: I0918 20:38:35.086764 21909 raft_consensus.cc:2644] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:35.087136 21909 raft_consensus.cc:472] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:35.087463 21909 raft_consensus.cc:2890] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:35.098723 21854 raft_consensus.cc:2311] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a2535df90af94322a63997ab490e9f7b for term 1 because replica is either leader or believes a valid leader to be alive. I0918 20:38:35.101925 21735 leader_election.cc:310] T 5effc580c4394916a52d24cfe1b086f2 P a2535df90af94322a63997ab490e9f7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2535df90af94322a63997ab490e9f7b; no voters: 5a30c225d1e543d69c2601bde2c33afc, 772cb6045c9d4a7785077cdfe1b5312a I0918 20:38:35.103368 21906 raft_consensus.cc:2592] T 5effc580c4394916a52d24cfe1b086f2 P a2535df90af94322a63997ab490e9f7b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0918 20:38:35.110002 21854 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f851d4a38fc246a1ad249ee5a6eee631" candidate_uuid: "5a30c225d1e543d69c2601bde2c33afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "772cb6045c9d4a7785077cdfe1b5312a" is_pre_election: true I0918 20:38:35.110715 21854 raft_consensus.cc:2350] T f851d4a38fc246a1ad249ee5a6eee631 P 772cb6045c9d4a7785077cdfe1b5312a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a30c225d1e543d69c2601bde2c33afc in term 0. I0918 20:38:35.109438 21499 catalog_manager.cc:4131] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a reported cstate change: term changed from 0 to 1, leader changed from to 772cb6045c9d4a7785077cdfe1b5312a (127.18.222.3). New cstate: current_term: 1 leader_uuid: "772cb6045c9d4a7785077cdfe1b5312a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } health_report { overall_health: HEALTHY } } } I0918 20:38:35.116070 21909 raft_consensus.cc:494] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } I0918 20:38:35.119315 21909 leader_election.cc:296] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc [CANDIDATE]: Term 1 election: Requested vote from peers a2535df90af94322a63997ab490e9f7b (127.18.222.2:43981), 772cb6045c9d4a7785077cdfe1b5312a (127.18.222.3:46835) I0918 20:38:35.120926 21854 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f851d4a38fc246a1ad249ee5a6eee631" candidate_uuid: "5a30c225d1e543d69c2601bde2c33afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "772cb6045c9d4a7785077cdfe1b5312a" I0918 20:38:35.121645 21854 raft_consensus.cc:2890] T f851d4a38fc246a1ad249ee5a6eee631 P 772cb6045c9d4a7785077cdfe1b5312a [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:35.127189 21772 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f851d4a38fc246a1ad249ee5a6eee631" candidate_uuid: "5a30c225d1e543d69c2601bde2c33afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2535df90af94322a63997ab490e9f7b" I0918 20:38:35.128520 21772 raft_consensus.cc:2890] T f851d4a38fc246a1ad249ee5a6eee631 P a2535df90af94322a63997ab490e9f7b [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:35.137568 21901 tablet_bootstrap.cc:439] T 56087a5c23f74746b0a391195509fb7a P 772cb6045c9d4a7785077cdfe1b5312a: No bootstrap required, opened a new log I0918 20:38:35.138222 21901 ts_tablet_manager.cc:1085] T 56087a5c23f74746b0a391195509fb7a P 772cb6045c9d4a7785077cdfe1b5312a: Time spent bootstrapping tablet: real 0.147s user 0.023s sys 0.013s I0918 20:38:35.139941 21901 raft_consensus.cc:340] T 56087a5c23f74746b0a391195509fb7a P 772cb6045c9d4a7785077cdfe1b5312a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } I0918 20:38:35.140455 21901 raft_consensus.cc:366] T 56087a5c23f74746b0a391195509fb7a P 772cb6045c9d4a7785077cdfe1b5312a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:35.140640 21901 raft_consensus.cc:705] T 56087a5c23f74746b0a391195509fb7a P 772cb6045c9d4a7785077cdfe1b5312a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 772cb6045c9d4a7785077cdfe1b5312a, State: Initialized, Role: FOLLOWER I0918 20:38:35.140836 21772 raft_consensus.cc:2350] T f851d4a38fc246a1ad249ee5a6eee631 P a2535df90af94322a63997ab490e9f7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a30c225d1e543d69c2601bde2c33afc in term 1. I0918 20:38:35.141208 21901 consensus_queue.cc:229] T 56087a5c23f74746b0a391195509fb7a P 772cb6045c9d4a7785077cdfe1b5312a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } I0918 20:38:35.143049 21901 ts_tablet_manager.cc:1113] T 56087a5c23f74746b0a391195509fb7a P 772cb6045c9d4a7785077cdfe1b5312a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s I0918 20:38:35.144734 21902 tablet_bootstrap.cc:439] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc: No bootstrap required, opened a new log I0918 20:38:35.145319 21902 ts_tablet_manager.cc:1085] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc: Time spent bootstrapping tablet: real 0.187s user 0.013s sys 0.005s I0918 20:38:35.147534 21902 raft_consensus.cc:340] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } I0918 20:38:35.151931 21902 raft_consensus.cc:366] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:35.152168 21902 raft_consensus.cc:705] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a30c225d1e543d69c2601bde2c33afc, State: Initialized, Role: FOLLOWER I0918 20:38:35.152647 21902 consensus_queue.cc:229] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } I0918 20:38:35.155071 21902 ts_tablet_manager.cc:1113] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s I0918 20:38:35.154546 21664 leader_election.cc:310] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a30c225d1e543d69c2601bde2c33afc, a2535df90af94322a63997ab490e9f7b; no voters: I0918 20:38:35.160023 21909 raft_consensus.cc:2644] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:35.168702 21909 raft_consensus.cc:667] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc [term 1 LEADER]: Becoming Leader. State: Replica: 5a30c225d1e543d69c2601bde2c33afc, State: Running, Role: LEADER I0918 20:38:35.169533 21909 consensus_queue.cc:206] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } I0918 20:38:35.156049 21854 raft_consensus.cc:2350] T f851d4a38fc246a1ad249ee5a6eee631 P 772cb6045c9d4a7785077cdfe1b5312a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a30c225d1e543d69c2601bde2c33afc in term 1. I0918 20:38:35.173806 21924 raft_consensus.cc:472] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:35.174381 21924 raft_consensus.cc:494] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } I0918 20:38:35.180780 21854 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "56087a5c23f74746b0a391195509fb7a" candidate_uuid: "5a30c225d1e543d69c2601bde2c33afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "772cb6045c9d4a7785077cdfe1b5312a" is_pre_election: true I0918 20:38:35.181684 21854 raft_consensus.cc:2350] T 56087a5c23f74746b0a391195509fb7a P 772cb6045c9d4a7785077cdfe1b5312a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a30c225d1e543d69c2601bde2c33afc in term 0. I0918 20:38:35.182812 21924 leader_election.cc:296] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 772cb6045c9d4a7785077cdfe1b5312a (127.18.222.3:46835), a2535df90af94322a63997ab490e9f7b (127.18.222.2:43981) I0918 20:38:35.182314 21772 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "56087a5c23f74746b0a391195509fb7a" candidate_uuid: "5a30c225d1e543d69c2601bde2c33afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2535df90af94322a63997ab490e9f7b" is_pre_election: true I0918 20:38:35.183363 21772 raft_consensus.cc:2350] T 56087a5c23f74746b0a391195509fb7a P a2535df90af94322a63997ab490e9f7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a30c225d1e543d69c2601bde2c33afc in term 0. I0918 20:38:35.183481 21664 leader_election.cc:310] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a30c225d1e543d69c2601bde2c33afc, 772cb6045c9d4a7785077cdfe1b5312a; no voters: I0918 20:38:35.184314 21924 raft_consensus.cc:2644] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:35.184531 21924 raft_consensus.cc:472] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:35.185250 21924 raft_consensus.cc:2890] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:35.227849 21924 raft_consensus.cc:494] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } I0918 20:38:35.232544 21924 leader_election.cc:296] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc [CANDIDATE]: Term 1 election: Requested vote from peers 772cb6045c9d4a7785077cdfe1b5312a (127.18.222.3:46835), a2535df90af94322a63997ab490e9f7b (127.18.222.2:43981) I0918 20:38:35.232725 21497 catalog_manager.cc:4131] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc reported cstate change: term changed from 0 to 1, leader changed from to 5a30c225d1e543d69c2601bde2c33afc (127.18.222.1). New cstate: current_term: 1 leader_uuid: "5a30c225d1e543d69c2601bde2c33afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } health_report { overall_health: UNKNOWN } } } I0918 20:38:35.234122 21772 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "56087a5c23f74746b0a391195509fb7a" candidate_uuid: "5a30c225d1e543d69c2601bde2c33afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2535df90af94322a63997ab490e9f7b" I0918 20:38:35.234707 21772 raft_consensus.cc:2890] T 56087a5c23f74746b0a391195509fb7a P a2535df90af94322a63997ab490e9f7b [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:35.242084 21854 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "56087a5c23f74746b0a391195509fb7a" candidate_uuid: "5a30c225d1e543d69c2601bde2c33afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "772cb6045c9d4a7785077cdfe1b5312a" I0918 20:38:35.242725 21854 raft_consensus.cc:2890] T 56087a5c23f74746b0a391195509fb7a P 772cb6045c9d4a7785077cdfe1b5312a [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:35.244073 21772 raft_consensus.cc:2350] T 56087a5c23f74746b0a391195509fb7a P a2535df90af94322a63997ab490e9f7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a30c225d1e543d69c2601bde2c33afc in term 1. I0918 20:38:35.245504 21664 leader_election.cc:310] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a30c225d1e543d69c2601bde2c33afc, a2535df90af94322a63997ab490e9f7b; no voters: I0918 20:38:35.247028 21924 raft_consensus.cc:2644] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:35.247498 21924 raft_consensus.cc:667] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc [term 1 LEADER]: Becoming Leader. State: Replica: 5a30c225d1e543d69c2601bde2c33afc, State: Running, Role: LEADER I0918 20:38:35.248991 21924 consensus_queue.cc:206] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } } peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } } I0918 20:38:35.249512 21854 raft_consensus.cc:2350] T 56087a5c23f74746b0a391195509fb7a P 772cb6045c9d4a7785077cdfe1b5312a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a30c225d1e543d69c2601bde2c33afc in term 1. I0918 20:38:35.273739 21497 catalog_manager.cc:4131] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc reported cstate change: term changed from 0 to 1, leader changed from to 5a30c225d1e543d69c2601bde2c33afc (127.18.222.1). New cstate: current_term: 1 leader_uuid: "5a30c225d1e543d69c2601bde2c33afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 } health_report { overall_health: HEALTHY } } } I0918 20:38:35.403029 19320 ksck_remote-test.cc:378] Generating row writes... I0918 20:38:35.494699 21772 raft_consensus.cc:1184] T 56087a5c23f74746b0a391195509fb7a P a2535df90af94322a63997ab490e9f7b [term 1 FOLLOWER]: Refusing update from remote peer 5a30c225d1e543d69c2601bde2c33afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:35.494954 21854 raft_consensus.cc:1184] T 56087a5c23f74746b0a391195509fb7a P 772cb6045c9d4a7785077cdfe1b5312a [term 1 FOLLOWER]: Refusing update from remote peer 5a30c225d1e543d69c2601bde2c33afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:35.497303 21909 consensus_queue.cc:984] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:35.498592 21909 consensus_queue.cc:984] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:35.502524 21853 raft_consensus.cc:1184] T f851d4a38fc246a1ad249ee5a6eee631 P 772cb6045c9d4a7785077cdfe1b5312a [term 1 FOLLOWER]: Refusing update from remote peer 5a30c225d1e543d69c2601bde2c33afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:35.504056 21924 consensus_queue.cc:984] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "772cb6045c9d4a7785077cdfe1b5312a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:35.514603 21771 raft_consensus.cc:1184] T f851d4a38fc246a1ad249ee5a6eee631 P a2535df90af94322a63997ab490e9f7b [term 1 FOLLOWER]: Refusing update from remote peer 5a30c225d1e543d69c2601bde2c33afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:35.516263 21924 consensus_queue.cc:984] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:35.578271 21907 consensus_queue.cc:984] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a30c225d1e543d69c2601bde2c33afc" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 39639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:35.593968 21770 raft_consensus.cc:1184] T 5effc580c4394916a52d24cfe1b086f2 P a2535df90af94322a63997ab490e9f7b [term 1 FOLLOWER]: Refusing update from remote peer 772cb6045c9d4a7785077cdfe1b5312a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:35.598711 21907 consensus_queue.cc:984] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2535df90af94322a63997ab490e9f7b" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:35.847860 21945 mvcc.cc:203] Tried to move safe_time back from 6425965017081356288 to 6425965015784914944. Current Snapshot: MvccSnapshot[committed={T|T < 6425965017081356288}] W0918 20:38:36.640327 21937 ksck_checksum.cc:285] failed to submit task to start additional tablet checksums: Service unavailable: The pool has been shut down. I0918 20:38:36.640776 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:36.747433 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:36.747865 19320 tablet_replica.cc:273] T 5effc580c4394916a52d24cfe1b086f2 P 5a30c225d1e543d69c2601bde2c33afc: stopping tablet replica I0918 20:38:36.748399 19320 raft_consensus.cc:2147] T 5effc580c4394916a52d24cfe1b086f2 P 5a30c225d1e543d69c2601bde2c33afc [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:36.748864 19320 raft_consensus.cc:2174] T 5effc580c4394916a52d24cfe1b086f2 P 5a30c225d1e543d69c2601bde2c33afc [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:36.754508 19320 tablet_replica.cc:273] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc: stopping tablet replica I0918 20:38:36.755187 19320 raft_consensus.cc:2147] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:36.756289 19320 raft_consensus.cc:2174] T 56087a5c23f74746b0a391195509fb7a P 5a30c225d1e543d69c2601bde2c33afc [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:36.759254 19320 tablet_replica.cc:273] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc: stopping tablet replica I0918 20:38:36.759876 19320 raft_consensus.cc:2147] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:36.760769 19320 raft_consensus.cc:2174] T f851d4a38fc246a1ad249ee5a6eee631 P 5a30c225d1e543d69c2601bde2c33afc [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:36.785181 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:36.785547 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:36.786151 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:36.806797 19320 tablet_server.cc:155] TabletServer shutting down... W0918 20:38:36.869606 21809 consensus_peers.cc:458] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a -> Peer 5a30c225d1e543d69c2601bde2c33afc (127.18.222.1:39639): Couldn't send request to peer 5a30c225d1e543d69c2601bde2c33afc. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.1:39639: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:38:36.884883 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:36.885345 19320 tablet_replica.cc:273] T f851d4a38fc246a1ad249ee5a6eee631 P a2535df90af94322a63997ab490e9f7b: stopping tablet replica I0918 20:38:36.885749 19320 raft_consensus.cc:2147] T f851d4a38fc246a1ad249ee5a6eee631 P a2535df90af94322a63997ab490e9f7b [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:36.886063 19320 raft_consensus.cc:2174] T f851d4a38fc246a1ad249ee5a6eee631 P a2535df90af94322a63997ab490e9f7b [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:36.889243 19320 tablet_replica.cc:273] T 56087a5c23f74746b0a391195509fb7a P a2535df90af94322a63997ab490e9f7b: stopping tablet replica I0918 20:38:36.889617 19320 raft_consensus.cc:2147] T 56087a5c23f74746b0a391195509fb7a P a2535df90af94322a63997ab490e9f7b [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:36.889928 19320 raft_consensus.cc:2174] T 56087a5c23f74746b0a391195509fb7a P a2535df90af94322a63997ab490e9f7b [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:36.893338 19320 tablet_replica.cc:273] T 5effc580c4394916a52d24cfe1b086f2 P a2535df90af94322a63997ab490e9f7b: stopping tablet replica I0918 20:38:36.893903 19320 raft_consensus.cc:2147] T 5effc580c4394916a52d24cfe1b086f2 P a2535df90af94322a63997ab490e9f7b [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:36.894312 19320 raft_consensus.cc:2174] T 5effc580c4394916a52d24cfe1b086f2 P a2535df90af94322a63997ab490e9f7b [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:36.914921 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:36.915177 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:36.915642 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:36.932430 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:36.974820 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:36.975131 19320 tablet_replica.cc:273] T 56087a5c23f74746b0a391195509fb7a P 772cb6045c9d4a7785077cdfe1b5312a: stopping tablet replica I0918 20:38:36.975519 19320 raft_consensus.cc:2147] T 56087a5c23f74746b0a391195509fb7a P 772cb6045c9d4a7785077cdfe1b5312a [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:36.975842 19320 raft_consensus.cc:2174] T 56087a5c23f74746b0a391195509fb7a P 772cb6045c9d4a7785077cdfe1b5312a [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:36.977823 19320 tablet_replica.cc:273] T f851d4a38fc246a1ad249ee5a6eee631 P 772cb6045c9d4a7785077cdfe1b5312a: stopping tablet replica I0918 20:38:36.978152 19320 raft_consensus.cc:2147] T f851d4a38fc246a1ad249ee5a6eee631 P 772cb6045c9d4a7785077cdfe1b5312a [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:36.978421 19320 raft_consensus.cc:2174] T f851d4a38fc246a1ad249ee5a6eee631 P 772cb6045c9d4a7785077cdfe1b5312a [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:36.980399 19320 tablet_replica.cc:273] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a: stopping tablet replica I0918 20:38:36.980724 19320 raft_consensus.cc:2147] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:36.981338 19320 raft_consensus.cc:2174] T 5effc580c4394916a52d24cfe1b086f2 P 772cb6045c9d4a7785077cdfe1b5312a [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:36.994107 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:36.994302 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:36.994663 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:37.004931 19320 master.cc:247] Master@127.18.222.62:40939 shutting down... I0918 20:38:37.031838 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:37.032582 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:37.032831 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 8eaf326bfbed476987e5379cbccca3c8: stopping tablet replica I0918 20:38:37.051785 19320 master.cc:258] Master@127.18.222.62:40939 shutdown complete. I0918 20:38:37.062535 19320 master.cc:247] Master@127.18.222.61:34847 shutting down... I0918 20:38:37.089802 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:37.090181 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:37.090412 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P c5165bb9c7334058a7bb869966482ee5: stopping tablet replica I0918 20:38:37.109241 19320 master.cc:258] Master@127.18.222.61:34847 shutdown complete. I0918 20:38:37.117511 19320 master.cc:247] Master@127.18.222.60:39895 shutting down... I0918 20:38:37.147872 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:37.148380 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:37.148702 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 23a446a27d414a34bbdab5f72442acfa: stopping tablet replica I0918 20:38:37.168479 19320 master.cc:258] Master@127.18.222.60:39895 shutdown complete. [ OK ] RemoteKsckTest.TestChecksum (3852 ms) [ RUN ] RemoteKsckTest.TestChecksumTimeout I0918 20:38:37.206246 19320 test_util.cc:212] Using random seed: -766872888 I0918 20:38:37.207372 19320 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.18.222.62:46257,127.18.222.61:41695,127.18.222.60:46633 I0918 20:38:37.231844 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 136092us I0918 20:38:37.232326 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:37.232542 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:37.232841 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:37.233017 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:37.234390 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/master-0-root/instance: uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" format_stamp: "Formatted at 2019-09-18 20:38:37 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:37.249662 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.015s user 0.013s sys 0.005s I0918 20:38:37.251889 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:37.264998 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0918 20:38:37.265192 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/master-0-root uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" format_stamp: "Formatted at 2019-09-18 20:38:37 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:37.265481 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:37.340093 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:46257 I0918 20:38:37.341873 19320 webserver.cc:374] Webserver started at http://127.18.222.62:43723/ using document root and password file I0918 20:38:37.344040 22052 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:37.351115 22052 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:37.362773 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 136092us I0918 20:38:37.363153 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:37.363327 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:37.363562 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:37.363687 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:37.365162 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/master-1-root/instance: uuid: "6feec5fc783e4905982503aeadbd8d16" format_stamp: "Formatted at 2019-09-18 20:38:37 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:37.370412 22052 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } has no permanent_uuid. Determining permanent_uuid... W0918 20:38:37.372499 22052 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:41695: Network error: Client connection negotiation failed: client connection to 127.18.222.61:41695: connect: Connection refused (error 111) I0918 20:38:37.377620 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.012s user 0.011s sys 0.000s I0918 20:38:37.379190 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:37.387151 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:38:37.387338 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/master-1-root uuid: "6feec5fc783e4905982503aeadbd8d16" format_stamp: "Formatted at 2019-09-18 20:38:37 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:37.387614 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:37.426920 22052 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } attempt: 1 W0918 20:38:37.429510 22052 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:41695: Network error: Client connection negotiation failed: client connection to 127.18.222.61:41695: connect: Connection refused (error 111) I0918 20:38:37.474947 22052 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } attempt: 2 W0918 20:38:37.477386 22052 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:41695: Network error: Client connection negotiation failed: client connection to 127.18.222.61:41695: connect: Connection refused (error 111) I0918 20:38:37.564095 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:41695 I0918 20:38:37.568115 19320 webserver.cc:374] Webserver started at http://127.18.222.61:32885/ using document root and password file I0918 20:38:37.572849 22109 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:37.581399 22109 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:37.587146 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 136092us I0918 20:38:37.587595 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:37.587832 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:37.588124 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:37.588263 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:37.589359 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/master-2-root/instance: uuid: "ab28e98d635942f098b2bcad0bfd652c" format_stamp: "Formatted at 2019-09-18 20:38:37 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:37.589810 22052 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } attempt: 3 I0918 20:38:37.597621 22109 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:37.608976 22052 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:37.609422 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.020s user 0.016s sys 0.004s W0918 20:38:37.611588 22052 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:46633: Network error: Client connection negotiation failed: client connection to 127.18.222.60:46633: connect: Connection refused (error 111) I0918 20:38:37.611835 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:37.624842 22109 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } has no permanent_uuid. Determining permanent_uuid... W0918 20:38:37.627655 22109 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:46633: Network error: Client connection negotiation failed: client connection to 127.18.222.60:46633: connect: Connection refused (error 111) I0918 20:38:37.628777 19320 fs_manager.cc:419] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s I0918 20:38:37.629007 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/master-2-root uuid: "ab28e98d635942f098b2bcad0bfd652c" format_stamp: "Formatted at 2019-09-18 20:38:37 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:37.629261 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:37.635074 22052 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } attempt: 1 W0918 20:38:37.637399 22052 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:46633: Network error: Client connection negotiation failed: client connection to 127.18.222.60:46633: connect: Connection refused (error 111) I0918 20:38:37.653270 22109 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } attempt: 1 W0918 20:38:37.655762 22109 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:46633: Network error: Client connection negotiation failed: client connection to 127.18.222.60:46633: connect: Connection refused (error 111) I0918 20:38:37.692927 22052 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } attempt: 2 W0918 20:38:37.695364 22052 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:46633: Network error: Client connection negotiation failed: client connection to 127.18.222.60:46633: connect: Connection refused (error 111) I0918 20:38:37.712235 22109 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } attempt: 2 W0918 20:38:37.714905 22109 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:46633: Network error: Client connection negotiation failed: client connection to 127.18.222.60:46633: connect: Connection refused (error 111) I0918 20:38:37.781742 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:46633 I0918 20:38:37.783602 19320 webserver.cc:374] Webserver started at http://127.18.222.60:36693/ using document root and password file I0918 20:38:37.783816 22052 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } attempt: 3 I0918 20:38:37.788203 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0918 20:38:37.789538 22167 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:37.799923 22167 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:37.806494 22052 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94: Bootstrap starting. I0918 20:38:37.809281 22052 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94: No blocks or log segments found. Creating new log. I0918 20:38:37.815397 22109 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } attempt: 3 I0918 20:38:37.817162 22052 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94: No bootstrap required, opened a new log I0918 20:38:37.819984 22052 raft_consensus.cc:340] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } I0918 20:38:37.820753 22052 raft_consensus.cc:366] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:37.820967 22052 raft_consensus.cc:705] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccffb4cc11b349b4ae52a0a13eb59b94, State: Initialized, Role: FOLLOWER I0918 20:38:37.821550 22052 consensus_queue.cc:229] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } I0918 20:38:37.823953 22167 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:37.824313 22052 sys_catalog.cc:340] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } } I0918 20:38:37.825232 22052 sys_catalog.cc:343] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:37.826485 22052 sys_catalog.cc:428] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:38:37.828764 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 I0918 20:38:37.826822 22171 sys_catalog.cc:340] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } } I0918 20:38:37.832806 22171 sys_catalog.cc:343] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94 [sys.catalog]: This master's current role is: FOLLOWER W0918 20:38:37.835474 22172 catalog_manager.cc:1155] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:38:37.835894 22172 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:37.842154 22167 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:37.844172 22109 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16: Bootstrap starting. I0918 20:38:37.847435 22109 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16: No blocks or log segments found. Creating new log. I0918 20:38:37.854459 22109 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16: No bootstrap required, opened a new log I0918 20:38:37.856940 22109 raft_consensus.cc:340] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } I0918 20:38:37.857729 22109 raft_consensus.cc:366] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:37.858005 22109 raft_consensus.cc:705] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6feec5fc783e4905982503aeadbd8d16, State: Initialized, Role: FOLLOWER I0918 20:38:37.858829 22109 consensus_queue.cc:229] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } I0918 20:38:37.863153 22167 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c: Bootstrap starting. I0918 20:38:37.864426 22174 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } } I0918 20:38:37.865203 22174 sys_catalog.cc:343] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:37.865751 22109 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } } I0918 20:38:37.866547 22109 sys_catalog.cc:343] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:37.867216 22167 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c: No blocks or log segments found. Creating new log. I0918 20:38:37.867725 22109 sys_catalog.cc:428] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:38:37.872184 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 W0918 20:38:37.873993 22175 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:38:37.874245 22175 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:37.875782 22167 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c: No bootstrap required, opened a new log I0918 20:38:37.877472 22167 raft_consensus.cc:340] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } I0918 20:38:37.877952 22167 raft_consensus.cc:366] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:37.878149 22167 raft_consensus.cc:705] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab28e98d635942f098b2bcad0bfd652c, State: Initialized, Role: FOLLOWER I0918 20:38:37.878636 22167 consensus_queue.cc:229] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } I0918 20:38:37.880057 22167 sys_catalog.cc:340] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } } I0918 20:38:37.880570 22167 sys_catalog.cc:343] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:37.880370 22177 sys_catalog.cc:340] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } } I0918 20:38:37.881211 22177 sys_catalog.cc:343] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:37.881495 22167 sys_catalog.cc:428] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c [sys.catalog]: configured and running, proceeding with master startup. I0918 20:38:37.882488 22178 catalog_manager.cc:569] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0918 20:38:37.910012 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 136092us I0918 20:38:37.910650 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:37.910887 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:37.911170 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:37.911342 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:37.912781 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/ts-0-root/instance: uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" format_stamp: "Formatted at 2019-09-18 20:38:37 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:37.928874 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.016s user 0.013s sys 0.004s I0918 20:38:37.931254 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:37.948976 19320 fs_manager.cc:419] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s I0918 20:38:37.949261 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/ts-0-root uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" format_stamp: "Formatted at 2019-09-18 20:38:37 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:37.949609 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:37.970504 22174 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:37.971014 22174 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } I0918 20:38:37.972493 22174 leader_election.cc:296] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccffb4cc11b349b4ae52a0a13eb59b94 (127.18.222.62:46257), ab28e98d635942f098b2bcad0bfd652c (127.18.222.60:46633) I0918 20:38:37.973510 22029 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6feec5fc783e4905982503aeadbd8d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" is_pre_election: true I0918 20:38:37.973899 22144 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6feec5fc783e4905982503aeadbd8d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab28e98d635942f098b2bcad0bfd652c" is_pre_election: true I0918 20:38:37.974278 22029 raft_consensus.cc:2350] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6feec5fc783e4905982503aeadbd8d16 in term 0. I0918 20:38:37.974756 22144 raft_consensus.cc:2350] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6feec5fc783e4905982503aeadbd8d16 in term 0. I0918 20:38:37.975517 22059 leader_election.cc:310] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6feec5fc783e4905982503aeadbd8d16, ccffb4cc11b349b4ae52a0a13eb59b94; no voters: I0918 20:38:37.976510 22174 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:37.976799 22174 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:37.976979 22174 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:37.980880 22174 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } I0918 20:38:37.983047 22029 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6feec5fc783e4905982503aeadbd8d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" I0918 20:38:37.983065 22144 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6feec5fc783e4905982503aeadbd8d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab28e98d635942f098b2bcad0bfd652c" I0918 20:38:37.983577 22174 leader_election.cc:296] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [CANDIDATE]: Term 1 election: Requested vote from peers ccffb4cc11b349b4ae52a0a13eb59b94 (127.18.222.62:46257), ab28e98d635942f098b2bcad0bfd652c (127.18.222.60:46633) I0918 20:38:37.983973 22144 raft_consensus.cc:2890] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:37.987776 22029 raft_consensus.cc:2890] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:37.991907 22144 raft_consensus.cc:2350] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6feec5fc783e4905982503aeadbd8d16 in term 1. I0918 20:38:37.991633 22029 raft_consensus.cc:2350] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6feec5fc783e4905982503aeadbd8d16 in term 1. I0918 20:38:37.993153 22060 leader_election.cc:310] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6feec5fc783e4905982503aeadbd8d16, ab28e98d635942f098b2bcad0bfd652c; no voters: I0918 20:38:37.994267 22174 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:37.996280 22174 raft_consensus.cc:667] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [term 1 LEADER]: Becoming Leader. State: Replica: 6feec5fc783e4905982503aeadbd8d16, State: Running, Role: LEADER I0918 20:38:37.996978 22174 consensus_queue.cc:206] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } I0918 20:38:38.001119 22185 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6feec5fc783e4905982503aeadbd8d16. Latest consensus state: current_term: 1 leader_uuid: "6feec5fc783e4905982503aeadbd8d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } } I0918 20:38:38.002172 22185 sys_catalog.cc:343] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [sys.catalog]: This master's current role is: LEADER I0918 20:38:38.004698 22187 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:38:38.012032 22187 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:38:38.024235 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:38.024493 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:38.105535 22144 raft_consensus.cc:1184] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c [term 1 FOLLOWER]: Refusing update from remote peer 6feec5fc783e4905982503aeadbd8d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:38.106461 22029 raft_consensus.cc:1184] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94 [term 1 FOLLOWER]: Refusing update from remote peer 6feec5fc783e4905982503aeadbd8d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:38.107244 22185 consensus_queue.cc:984] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:38.108346 22174 consensus_queue.cc:984] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:38.119598 22177 sys_catalog.cc:340] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6feec5fc783e4905982503aeadbd8d16. Latest consensus state: current_term: 1 leader_uuid: "6feec5fc783e4905982503aeadbd8d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } } I0918 20:38:38.122663 22177 sys_catalog.cc:343] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:38.138958 22171 sys_catalog.cc:340] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6feec5fc783e4905982503aeadbd8d16. Latest consensus state: current_term: 1 leader_uuid: "6feec5fc783e4905982503aeadbd8d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } } I0918 20:38:38.140517 22171 sys_catalog.cc:343] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:38.156695 22174 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6feec5fc783e4905982503aeadbd8d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } } I0918 20:38:38.158197 22171 sys_catalog.cc:340] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6feec5fc783e4905982503aeadbd8d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } } I0918 20:38:38.158802 22174 sys_catalog.cc:343] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [sys.catalog]: This master's current role is: LEADER I0918 20:38:38.159183 22171 sys_catalog.cc:343] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:38.158771 22177 sys_catalog.cc:340] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6feec5fc783e4905982503aeadbd8d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } } I0918 20:38:38.159657 22177 sys_catalog.cc:343] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:38.159734 22174 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6feec5fc783e4905982503aeadbd8d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccffb4cc11b349b4ae52a0a13eb59b94" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 46257 } } peers { permanent_uuid: "6feec5fc783e4905982503aeadbd8d16" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 41695 } } peers { permanent_uuid: "ab28e98d635942f098b2bcad0bfd652c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46633 } } } I0918 20:38:38.161509 22174 sys_catalog.cc:343] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [sys.catalog]: This master's current role is: LEADER I0918 20:38:38.167187 22187 catalog_manager.cc:981] Generated new certificate authority record I0918 20:38:38.169333 22187 catalog_manager.cc:1106] Loading token signing keys... I0918 20:38:38.194310 22187 catalog_manager.cc:4446] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16: Generated new TSK 0 I0918 20:38:38.195056 22187 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:38:38.212896 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:35785 I0918 20:38:38.215538 19320 webserver.cc:374] Webserver started at http://127.18.222.1:41949/ using document root and password file I0918 20:38:38.249894 22255 heartbeater.cc:322] Connected to a master server at 127.18.222.61:41695 I0918 20:38:38.250242 22255 heartbeater.cc:406] Registering TS with master... I0918 20:38:38.250731 22255 heartbeater.cc:455] Master 127.18.222.61:41695 requested a full tablet report, sending... I0918 20:38:38.254247 22076 ts_manager.cc:186] Registered new tserver with Master: b4bb3d9a778a4d8e9a3987f0a6d863b0 (127.18.222.1:35785) I0918 20:38:38.254464 22256 heartbeater.cc:322] Connected to a master server at 127.18.222.60:46633 I0918 20:38:38.254838 22256 heartbeater.cc:406] Registering TS with master... I0918 20:38:38.256323 22256 heartbeater.cc:455] Master 127.18.222.60:46633 requested a full tablet report, sending... I0918 20:38:38.257975 22076 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44600 I0918 20:38:38.258973 22134 ts_manager.cc:186] Registered new tserver with Master: b4bb3d9a778a4d8e9a3987f0a6d863b0 (127.18.222.1:35785) I0918 20:38:38.264392 22257 heartbeater.cc:322] Connected to a master server at 127.18.222.62:46257 I0918 20:38:38.264699 22257 heartbeater.cc:406] Registering TS with master... I0918 20:38:38.265058 22257 heartbeater.cc:455] Master 127.18.222.62:46257 requested a full tablet report, sending... I0918 20:38:38.267227 22019 ts_manager.cc:186] Registered new tserver with Master: b4bb3d9a778a4d8e9a3987f0a6d863b0 (127.18.222.1:35785) I0918 20:38:38.269086 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 136592us I0918 20:38:38.269553 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:38.269767 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:38.270062 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:38.270241 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:38.271612 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/ts-1-root/instance: uuid: "1197af9577844861bc2218a92e37c173" format_stamp: "Formatted at 2019-09-18 20:38:38 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:38.272354 22255 heartbeater.cc:447] Master 127.18.222.61:41695 was elected leader, sending a full tablet report... I0918 20:38:38.291033 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.019s user 0.014s sys 0.005s I0918 20:38:38.293489 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:38.309486 19320 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s I0918 20:38:38.309773 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/ts-1-root uuid: "1197af9577844861bc2218a92e37c173" format_stamp: "Formatted at 2019-09-18 20:38:38 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:38.310104 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:38.359648 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:38.360100 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:38.456074 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.2:35363 I0918 20:38:38.460096 19320 webserver.cc:374] Webserver started at http://127.18.222.2:45015/ using document root and password file I0918 20:38:38.506549 22326 heartbeater.cc:322] Connected to a master server at 127.18.222.61:41695 I0918 20:38:38.506948 22326 heartbeater.cc:406] Registering TS with master... I0918 20:38:38.507382 22326 heartbeater.cc:455] Master 127.18.222.61:41695 requested a full tablet report, sending... I0918 20:38:38.509522 22076 ts_manager.cc:186] Registered new tserver with Master: 1197af9577844861bc2218a92e37c173 (127.18.222.2:35363) I0918 20:38:38.517081 22076 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44606 I0918 20:38:38.522084 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 136592us I0918 20:38:38.522728 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:38.523044 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:38.523416 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:38.523660 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:38.525156 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/ts-2-root/instance: uuid: "2f1a504155064a55b5a716a520e1f2ac" format_stamp: "Formatted at 2019-09-18 20:38:38 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:38.526912 22329 heartbeater.cc:322] Connected to a master server at 127.18.222.62:46257 I0918 20:38:38.527302 22329 heartbeater.cc:406] Registering TS with master... I0918 20:38:38.527915 22329 heartbeater.cc:455] Master 127.18.222.62:46257 requested a full tablet report, sending... I0918 20:38:38.530156 22327 heartbeater.cc:322] Connected to a master server at 127.18.222.60:46633 I0918 20:38:38.530580 22327 heartbeater.cc:406] Registering TS with master... I0918 20:38:38.531108 22327 heartbeater.cc:455] Master 127.18.222.60:46633 requested a full tablet report, sending... I0918 20:38:38.531989 22326 heartbeater.cc:447] Master 127.18.222.61:41695 was elected leader, sending a full tablet report... I0918 20:38:38.537029 22134 ts_manager.cc:186] Registered new tserver with Master: 1197af9577844861bc2218a92e37c173 (127.18.222.2:35363) I0918 20:38:38.537178 22018 ts_manager.cc:186] Registered new tserver with Master: 1197af9577844861bc2218a92e37c173 (127.18.222.2:35363) I0918 20:38:38.542945 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.017s user 0.016s sys 0.000s I0918 20:38:38.545063 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:38.559342 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0918 20:38:38.559757 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/ts-2-root uuid: "2f1a504155064a55b5a716a520e1f2ac" format_stamp: "Formatted at 2019-09-18 20:38:38 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:38.560173 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1568839089958813-19320/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:38.640213 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:38.640450 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:38.832772 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.3:46347 I0918 20:38:38.836872 19320 webserver.cc:374] Webserver started at http://127.18.222.3:44389/ using document root and password file I0918 20:38:38.843572 22172 catalog_manager.cc:1153] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94: acquiring CA information for follower catalog manager: success I0918 20:38:38.865413 22172 catalog_manager.cc:1181] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:38:38.896657 22178 catalog_manager.cc:1153] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c: acquiring CA information for follower catalog manager: success I0918 20:38:38.900806 22178 catalog_manager.cc:1181] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:38:38.907599 22397 heartbeater.cc:322] Connected to a master server at 127.18.222.61:41695 I0918 20:38:38.907991 22397 heartbeater.cc:406] Registering TS with master... I0918 20:38:38.908532 22397 heartbeater.cc:455] Master 127.18.222.61:41695 requested a full tablet report, sending... I0918 20:38:38.910519 22076 ts_manager.cc:186] Registered new tserver with Master: 2f1a504155064a55b5a716a520e1f2ac (127.18.222.3:46347) I0918 20:38:38.918897 22076 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44612 I0918 20:38:38.930156 22398 heartbeater.cc:322] Connected to a master server at 127.18.222.60:46633 I0918 20:38:38.930550 22398 heartbeater.cc:406] Registering TS with master... I0918 20:38:38.931090 22398 heartbeater.cc:455] Master 127.18.222.60:46633 requested a full tablet report, sending... I0918 20:38:38.932863 22397 heartbeater.cc:447] Master 127.18.222.61:41695 was elected leader, sending a full tablet report... I0918 20:38:38.933774 22133 ts_manager.cc:186] Registered new tserver with Master: 2f1a504155064a55b5a716a520e1f2ac (127.18.222.3:46347) I0918 20:38:38.946434 22400 heartbeater.cc:322] Connected to a master server at 127.18.222.62:46257 I0918 20:38:38.946811 22400 heartbeater.cc:406] Registering TS with master... I0918 20:38:38.947263 22400 heartbeater.cc:455] Master 127.18.222.62:46257 requested a full tablet report, sending... I0918 20:38:38.949947 22019 ts_manager.cc:186] Registered new tserver with Master: 2f1a504155064a55b5a716a520e1f2ac (127.18.222.3:46347) I0918 20:38:38.951303 19320 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.047464377s I0918 20:38:39.050997 22076 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44620: name: "ksck-test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\n\000\000\000\004\001\024\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:38:39.053042 22076 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:38:39.239224 22293 tablet_service.cc:1022] Processing CreateTablet for tablet ba7b5078d5404513ad4b6aa39b249244 (table=ksck-test-table [id=9419bad4fcb24e2eb5721b8ad06cae1e]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:39.242456 22293 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet ba7b5078d5404513ad4b6aa39b249244. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:39.253803 22291 tablet_service.cc:1022] Processing CreateTablet for tablet 517a906cb7664795939d3758fc7d7a17 (table=ksck-test-table [id=9419bad4fcb24e2eb5721b8ad06cae1e]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:39.254290 22292 tablet_service.cc:1022] Processing CreateTablet for tablet bd65e74409ad46799a393e20fbc31c54 (table=ksck-test-table [id=9419bad4fcb24e2eb5721b8ad06cae1e]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:39.254963 22291 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 517a906cb7664795939d3758fc7d7a17. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:39.255473 22292 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet bd65e74409ad46799a393e20fbc31c54. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:39.304579 22426 tablet_bootstrap.cc:439] T ba7b5078d5404513ad4b6aa39b249244 P 1197af9577844861bc2218a92e37c173: Bootstrap starting. I0918 20:38:39.310706 22426 tablet_bootstrap.cc:590] T ba7b5078d5404513ad4b6aa39b249244 P 1197af9577844861bc2218a92e37c173: No blocks or log segments found. Creating new log. I0918 20:38:39.320113 22213 tablet_service.cc:1022] Processing CreateTablet for tablet ba7b5078d5404513ad4b6aa39b249244 (table=ksck-test-table [id=9419bad4fcb24e2eb5721b8ad06cae1e]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:39.321331 22212 tablet_service.cc:1022] Processing CreateTablet for tablet 517a906cb7664795939d3758fc7d7a17 (table=ksck-test-table [id=9419bad4fcb24e2eb5721b8ad06cae1e]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:39.321802 22213 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet ba7b5078d5404513ad4b6aa39b249244. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:39.322958 22211 tablet_service.cc:1022] Processing CreateTablet for tablet bd65e74409ad46799a393e20fbc31c54 (table=ksck-test-table [id=9419bad4fcb24e2eb5721b8ad06cae1e]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:39.324245 22212 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 517a906cb7664795939d3758fc7d7a17. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:39.328688 22211 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet bd65e74409ad46799a393e20fbc31c54. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:39.331095 22364 tablet_service.cc:1022] Processing CreateTablet for tablet ba7b5078d5404513ad4b6aa39b249244 (table=ksck-test-table [id=9419bad4fcb24e2eb5721b8ad06cae1e]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:39.332422 22364 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet ba7b5078d5404513ad4b6aa39b249244. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:39.341711 22363 tablet_service.cc:1022] Processing CreateTablet for tablet 517a906cb7664795939d3758fc7d7a17 (table=ksck-test-table [id=9419bad4fcb24e2eb5721b8ad06cae1e]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:39.342826 22363 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 517a906cb7664795939d3758fc7d7a17. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:39.345799 22362 tablet_service.cc:1022] Processing CreateTablet for tablet bd65e74409ad46799a393e20fbc31c54 (table=ksck-test-table [id=9419bad4fcb24e2eb5721b8ad06cae1e]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:39.346829 22362 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet bd65e74409ad46799a393e20fbc31c54. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:39.350905 22426 tablet_bootstrap.cc:439] T ba7b5078d5404513ad4b6aa39b249244 P 1197af9577844861bc2218a92e37c173: No bootstrap required, opened a new log I0918 20:38:39.352417 22426 ts_tablet_manager.cc:1085] T ba7b5078d5404513ad4b6aa39b249244 P 1197af9577844861bc2218a92e37c173: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.000s I0918 20:38:39.363504 22426 raft_consensus.cc:340] T ba7b5078d5404513ad4b6aa39b249244 P 1197af9577844861bc2218a92e37c173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } I0918 20:38:39.368803 22426 raft_consensus.cc:366] T ba7b5078d5404513ad4b6aa39b249244 P 1197af9577844861bc2218a92e37c173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:39.369096 22426 raft_consensus.cc:705] T ba7b5078d5404513ad4b6aa39b249244 P 1197af9577844861bc2218a92e37c173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1197af9577844861bc2218a92e37c173, State: Initialized, Role: FOLLOWER I0918 20:38:39.370507 22428 tablet_bootstrap.cc:439] T bd65e74409ad46799a393e20fbc31c54 P 2f1a504155064a55b5a716a520e1f2ac: Bootstrap starting. I0918 20:38:39.374914 22428 tablet_bootstrap.cc:590] T bd65e74409ad46799a393e20fbc31c54 P 2f1a504155064a55b5a716a520e1f2ac: No blocks or log segments found. Creating new log. I0918 20:38:39.372097 22426 consensus_queue.cc:229] T ba7b5078d5404513ad4b6aa39b249244 P 1197af9577844861bc2218a92e37c173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } I0918 20:38:39.387357 22426 ts_tablet_manager.cc:1113] T ba7b5078d5404513ad4b6aa39b249244 P 1197af9577844861bc2218a92e37c173: Time spent starting tablet: real 0.034s user 0.006s sys 0.000s I0918 20:38:39.397933 22430 tablet_bootstrap.cc:439] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0: Bootstrap starting. I0918 20:38:39.401841 22428 tablet_bootstrap.cc:439] T bd65e74409ad46799a393e20fbc31c54 P 2f1a504155064a55b5a716a520e1f2ac: No bootstrap required, opened a new log I0918 20:38:39.402494 22428 ts_tablet_manager.cc:1085] T bd65e74409ad46799a393e20fbc31c54 P 2f1a504155064a55b5a716a520e1f2ac: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.004s I0918 20:38:39.404736 22428 raft_consensus.cc:340] T bd65e74409ad46799a393e20fbc31c54 P 2f1a504155064a55b5a716a520e1f2ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } I0918 20:38:39.405751 22428 raft_consensus.cc:366] T bd65e74409ad46799a393e20fbc31c54 P 2f1a504155064a55b5a716a520e1f2ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:39.406033 22428 raft_consensus.cc:705] T bd65e74409ad46799a393e20fbc31c54 P 2f1a504155064a55b5a716a520e1f2ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f1a504155064a55b5a716a520e1f2ac, State: Initialized, Role: FOLLOWER I0918 20:38:39.406601 22428 consensus_queue.cc:229] T bd65e74409ad46799a393e20fbc31c54 P 2f1a504155064a55b5a716a520e1f2ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } I0918 20:38:39.395972 22426 tablet_bootstrap.cc:439] T 517a906cb7664795939d3758fc7d7a17 P 1197af9577844861bc2218a92e37c173: Bootstrap starting. I0918 20:38:39.408149 22430 tablet_bootstrap.cc:590] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0: No blocks or log segments found. Creating new log. I0918 20:38:39.414520 22426 tablet_bootstrap.cc:590] T 517a906cb7664795939d3758fc7d7a17 P 1197af9577844861bc2218a92e37c173: No blocks or log segments found. Creating new log. I0918 20:38:39.447376 22430 tablet_bootstrap.cc:439] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0: No bootstrap required, opened a new log I0918 20:38:39.451376 22430 ts_tablet_manager.cc:1085] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0: Time spent bootstrapping tablet: real 0.056s user 0.020s sys 0.006s I0918 20:38:39.453747 22430 raft_consensus.cc:340] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } I0918 20:38:39.454612 22430 raft_consensus.cc:366] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:39.454905 22430 raft_consensus.cc:705] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bb3d9a778a4d8e9a3987f0a6d863b0, State: Initialized, Role: FOLLOWER I0918 20:38:39.455751 22430 consensus_queue.cc:229] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } I0918 20:38:39.470118 22428 ts_tablet_manager.cc:1113] T bd65e74409ad46799a393e20fbc31c54 P 2f1a504155064a55b5a716a520e1f2ac: Time spent starting tablet: real 0.067s user 0.013s sys 0.017s I0918 20:38:39.473096 22428 tablet_bootstrap.cc:439] T ba7b5078d5404513ad4b6aa39b249244 P 2f1a504155064a55b5a716a520e1f2ac: Bootstrap starting. I0918 20:38:39.472021 22430 ts_tablet_manager.cc:1113] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0: Time spent starting tablet: real 0.020s user 0.003s sys 0.006s I0918 20:38:39.476681 22428 tablet_bootstrap.cc:590] T ba7b5078d5404513ad4b6aa39b249244 P 2f1a504155064a55b5a716a520e1f2ac: No blocks or log segments found. Creating new log. I0918 20:38:39.478022 22430 tablet_bootstrap.cc:439] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0: Bootstrap starting. I0918 20:38:39.482245 22430 tablet_bootstrap.cc:590] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0: No blocks or log segments found. Creating new log. I0918 20:38:39.488549 22426 tablet_bootstrap.cc:439] T 517a906cb7664795939d3758fc7d7a17 P 1197af9577844861bc2218a92e37c173: No bootstrap required, opened a new log I0918 20:38:39.489718 22426 ts_tablet_manager.cc:1085] T 517a906cb7664795939d3758fc7d7a17 P 1197af9577844861bc2218a92e37c173: Time spent bootstrapping tablet: real 0.096s user 0.002s sys 0.006s I0918 20:38:39.494424 22428 tablet_bootstrap.cc:439] T ba7b5078d5404513ad4b6aa39b249244 P 2f1a504155064a55b5a716a520e1f2ac: No bootstrap required, opened a new log I0918 20:38:39.495167 22428 ts_tablet_manager.cc:1085] T ba7b5078d5404513ad4b6aa39b249244 P 2f1a504155064a55b5a716a520e1f2ac: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.003s I0918 20:38:39.496079 22426 raft_consensus.cc:340] T 517a906cb7664795939d3758fc7d7a17 P 1197af9577844861bc2218a92e37c173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } I0918 20:38:39.500308 22426 raft_consensus.cc:366] T 517a906cb7664795939d3758fc7d7a17 P 1197af9577844861bc2218a92e37c173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:39.500507 22426 raft_consensus.cc:705] T 517a906cb7664795939d3758fc7d7a17 P 1197af9577844861bc2218a92e37c173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1197af9577844861bc2218a92e37c173, State: Initialized, Role: FOLLOWER I0918 20:38:39.501169 22426 consensus_queue.cc:229] T 517a906cb7664795939d3758fc7d7a17 P 1197af9577844861bc2218a92e37c173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } I0918 20:38:39.505290 22428 raft_consensus.cc:340] T ba7b5078d5404513ad4b6aa39b249244 P 2f1a504155064a55b5a716a520e1f2ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } I0918 20:38:39.505848 22428 raft_consensus.cc:366] T ba7b5078d5404513ad4b6aa39b249244 P 2f1a504155064a55b5a716a520e1f2ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:39.506026 22428 raft_consensus.cc:705] T ba7b5078d5404513ad4b6aa39b249244 P 2f1a504155064a55b5a716a520e1f2ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f1a504155064a55b5a716a520e1f2ac, State: Initialized, Role: FOLLOWER I0918 20:38:39.506469 22428 consensus_queue.cc:229] T ba7b5078d5404513ad4b6aa39b249244 P 2f1a504155064a55b5a716a520e1f2ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } I0918 20:38:39.516383 22426 ts_tablet_manager.cc:1113] T 517a906cb7664795939d3758fc7d7a17 P 1197af9577844861bc2218a92e37c173: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s I0918 20:38:39.521533 22426 tablet_bootstrap.cc:439] T bd65e74409ad46799a393e20fbc31c54 P 1197af9577844861bc2218a92e37c173: Bootstrap starting. I0918 20:38:39.525256 22430 tablet_bootstrap.cc:439] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0: No bootstrap required, opened a new log I0918 20:38:39.525982 22430 ts_tablet_manager.cc:1085] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0: Time spent bootstrapping tablet: real 0.048s user 0.007s sys 0.002s I0918 20:38:39.525482 22426 tablet_bootstrap.cc:590] T bd65e74409ad46799a393e20fbc31c54 P 1197af9577844861bc2218a92e37c173: No blocks or log segments found. Creating new log. I0918 20:38:39.535848 22428 ts_tablet_manager.cc:1113] T ba7b5078d5404513ad4b6aa39b249244 P 2f1a504155064a55b5a716a520e1f2ac: Time spent starting tablet: real 0.039s user 0.005s sys 0.000s I0918 20:38:39.536808 22428 tablet_bootstrap.cc:439] T 517a906cb7664795939d3758fc7d7a17 P 2f1a504155064a55b5a716a520e1f2ac: Bootstrap starting. I0918 20:38:39.537761 22430 raft_consensus.cc:340] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } I0918 20:38:39.538854 22430 raft_consensus.cc:366] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:39.539137 22430 raft_consensus.cc:705] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bb3d9a778a4d8e9a3987f0a6d863b0, State: Initialized, Role: FOLLOWER I0918 20:38:39.544797 22430 consensus_queue.cc:229] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } I0918 20:38:39.552373 22430 ts_tablet_manager.cc:1113] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s I0918 20:38:39.557073 22430 tablet_bootstrap.cc:439] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0: Bootstrap starting. I0918 20:38:39.560613 22430 tablet_bootstrap.cc:590] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0: No blocks or log segments found. Creating new log. I0918 20:38:39.577775 22426 tablet_bootstrap.cc:439] T bd65e74409ad46799a393e20fbc31c54 P 1197af9577844861bc2218a92e37c173: No bootstrap required, opened a new log I0918 20:38:39.580539 22430 tablet_bootstrap.cc:439] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0: No bootstrap required, opened a new log I0918 20:38:39.581115 22430 ts_tablet_manager.cc:1085] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.005s I0918 20:38:39.583089 22426 ts_tablet_manager.cc:1085] T bd65e74409ad46799a393e20fbc31c54 P 1197af9577844861bc2218a92e37c173: Time spent bootstrapping tablet: real 0.062s user 0.030s sys 0.008s I0918 20:38:39.585147 22428 tablet_bootstrap.cc:590] T 517a906cb7664795939d3758fc7d7a17 P 2f1a504155064a55b5a716a520e1f2ac: No blocks or log segments found. Creating new log. I0918 20:38:39.589424 22430 raft_consensus.cc:340] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } I0918 20:38:39.590255 22430 raft_consensus.cc:366] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:39.590498 22430 raft_consensus.cc:705] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bb3d9a778a4d8e9a3987f0a6d863b0, State: Initialized, Role: FOLLOWER I0918 20:38:39.591080 22430 consensus_queue.cc:229] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } I0918 20:38:39.592000 22434 raft_consensus.cc:472] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:39.592478 22434 raft_consensus.cc:494] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } I0918 20:38:39.593837 22428 tablet_bootstrap.cc:439] T 517a906cb7664795939d3758fc7d7a17 P 2f1a504155064a55b5a716a520e1f2ac: No bootstrap required, opened a new log I0918 20:38:39.594483 22434 leader_election.cc:296] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1197af9577844861bc2218a92e37c173 (127.18.222.2:35363), 2f1a504155064a55b5a716a520e1f2ac (127.18.222.3:46347) I0918 20:38:39.597687 22429 raft_consensus.cc:472] T ba7b5078d5404513ad4b6aa39b249244 P 1197af9577844861bc2218a92e37c173 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:39.598174 22429 raft_consensus.cc:494] T ba7b5078d5404513ad4b6aa39b249244 P 1197af9577844861bc2218a92e37c173 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } I0918 20:38:39.600436 22428 ts_tablet_manager.cc:1085] T 517a906cb7664795939d3758fc7d7a17 P 2f1a504155064a55b5a716a520e1f2ac: Time spent bootstrapping tablet: real 0.064s user 0.007s sys 0.000s I0918 20:38:39.590101 22426 raft_consensus.cc:340] T bd65e74409ad46799a393e20fbc31c54 P 1197af9577844861bc2218a92e37c173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } I0918 20:38:39.614264 22426 raft_consensus.cc:366] T bd65e74409ad46799a393e20fbc31c54 P 1197af9577844861bc2218a92e37c173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:39.614568 22426 raft_consensus.cc:705] T bd65e74409ad46799a393e20fbc31c54 P 1197af9577844861bc2218a92e37c173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1197af9577844861bc2218a92e37c173, State: Initialized, Role: FOLLOWER I0918 20:38:39.621673 22429 leader_election.cc:296] T ba7b5078d5404513ad4b6aa39b249244 P 1197af9577844861bc2218a92e37c173 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f1a504155064a55b5a716a520e1f2ac (127.18.222.3:46347), b4bb3d9a778a4d8e9a3987f0a6d863b0 (127.18.222.1:35785) I0918 20:38:39.615389 22426 consensus_queue.cc:229] T bd65e74409ad46799a393e20fbc31c54 P 1197af9577844861bc2218a92e37c173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } I0918 20:38:39.651314 22434 raft_consensus.cc:472] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:39.652245 22426 ts_tablet_manager.cc:1113] T bd65e74409ad46799a393e20fbc31c54 P 1197af9577844861bc2218a92e37c173: Time spent starting tablet: real 0.069s user 0.004s sys 0.003s I0918 20:38:39.652060 22434 raft_consensus.cc:494] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } I0918 20:38:39.654321 22434 leader_election.cc:296] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1197af9577844861bc2218a92e37c173 (127.18.222.2:35363), 2f1a504155064a55b5a716a520e1f2ac (127.18.222.3:46347) I0918 20:38:39.682253 22430 ts_tablet_manager.cc:1113] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0: Time spent starting tablet: real 0.101s user 0.039s sys 0.018s I0918 20:38:39.694630 22441 raft_consensus.cc:472] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:39.696225 22441 raft_consensus.cc:494] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } I0918 20:38:39.698894 22441 leader_election.cc:296] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1197af9577844861bc2218a92e37c173 (127.18.222.2:35363), 2f1a504155064a55b5a716a520e1f2ac (127.18.222.3:46347) I0918 20:38:39.691028 22428 raft_consensus.cc:340] T 517a906cb7664795939d3758fc7d7a17 P 2f1a504155064a55b5a716a520e1f2ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } I0918 20:38:39.701802 22428 raft_consensus.cc:366] T 517a906cb7664795939d3758fc7d7a17 P 2f1a504155064a55b5a716a520e1f2ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:39.703799 22428 raft_consensus.cc:705] T 517a906cb7664795939d3758fc7d7a17 P 2f1a504155064a55b5a716a520e1f2ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f1a504155064a55b5a716a520e1f2ac, State: Initialized, Role: FOLLOWER I0918 20:38:39.707564 22303 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ba7b5078d5404513ad4b6aa39b249244" candidate_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1197af9577844861bc2218a92e37c173" is_pre_election: true I0918 20:38:39.707917 22302 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bd65e74409ad46799a393e20fbc31c54" candidate_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1197af9577844861bc2218a92e37c173" is_pre_election: true I0918 20:38:39.708295 22301 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "517a906cb7664795939d3758fc7d7a17" candidate_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1197af9577844861bc2218a92e37c173" is_pre_election: true I0918 20:38:39.709309 22303 raft_consensus.cc:2350] T ba7b5078d5404513ad4b6aa39b249244 P 1197af9577844861bc2218a92e37c173 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4bb3d9a778a4d8e9a3987f0a6d863b0 in term 0. I0918 20:38:39.709674 22301 raft_consensus.cc:2350] T 517a906cb7664795939d3758fc7d7a17 P 1197af9577844861bc2218a92e37c173 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4bb3d9a778a4d8e9a3987f0a6d863b0 in term 0. I0918 20:38:39.709358 22302 raft_consensus.cc:2350] T bd65e74409ad46799a393e20fbc31c54 P 1197af9577844861bc2218a92e37c173 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4bb3d9a778a4d8e9a3987f0a6d863b0 in term 0. I0918 20:38:39.712281 22181 leader_election.cc:310] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1197af9577844861bc2218a92e37c173, b4bb3d9a778a4d8e9a3987f0a6d863b0; no voters: I0918 20:38:39.713263 22181 leader_election.cc:310] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1197af9577844861bc2218a92e37c173, b4bb3d9a778a4d8e9a3987f0a6d863b0; no voters: I0918 20:38:39.714056 22434 raft_consensus.cc:2644] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:39.714386 22434 raft_consensus.cc:472] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:39.713801 22428 consensus_queue.cc:229] T 517a906cb7664795939d3758fc7d7a17 P 2f1a504155064a55b5a716a520e1f2ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } I0918 20:38:39.714629 22434 raft_consensus.cc:2890] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:39.714948 22441 raft_consensus.cc:2644] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:39.715281 22441 raft_consensus.cc:472] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:39.717876 22374 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ba7b5078d5404513ad4b6aa39b249244" candidate_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f1a504155064a55b5a716a520e1f2ac" is_pre_election: true I0918 20:38:39.718673 22374 raft_consensus.cc:2350] T ba7b5078d5404513ad4b6aa39b249244 P 2f1a504155064a55b5a716a520e1f2ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4bb3d9a778a4d8e9a3987f0a6d863b0 in term 0. I0918 20:38:39.718911 22181 leader_election.cc:310] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1197af9577844861bc2218a92e37c173, b4bb3d9a778a4d8e9a3987f0a6d863b0; no voters: I0918 20:38:39.715629 22441 raft_consensus.cc:2890] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:39.722688 22373 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "517a906cb7664795939d3758fc7d7a17" candidate_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f1a504155064a55b5a716a520e1f2ac" is_pre_election: true I0918 20:38:39.723408 22373 raft_consensus.cc:2350] T 517a906cb7664795939d3758fc7d7a17 P 2f1a504155064a55b5a716a520e1f2ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4bb3d9a778a4d8e9a3987f0a6d863b0 in term 0. I0918 20:38:39.720978 22374 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bd65e74409ad46799a393e20fbc31c54" candidate_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f1a504155064a55b5a716a520e1f2ac" is_pre_election: true I0918 20:38:39.722995 22428 ts_tablet_manager.cc:1113] T 517a906cb7664795939d3758fc7d7a17 P 2f1a504155064a55b5a716a520e1f2ac: Time spent starting tablet: real 0.122s user 0.005s sys 0.001s I0918 20:38:39.724202 22374 raft_consensus.cc:2350] T bd65e74409ad46799a393e20fbc31c54 P 2f1a504155064a55b5a716a520e1f2ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4bb3d9a778a4d8e9a3987f0a6d863b0 in term 0. I0918 20:38:39.729290 22434 raft_consensus.cc:494] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } I0918 20:38:39.730932 22434 leader_election.cc:296] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [CANDIDATE]: Term 1 election: Requested vote from peers 1197af9577844861bc2218a92e37c173 (127.18.222.2:35363), 2f1a504155064a55b5a716a520e1f2ac (127.18.222.3:46347) I0918 20:38:39.731304 22434 raft_consensus.cc:2644] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:39.733412 22374 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ba7b5078d5404513ad4b6aa39b249244" candidate_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f1a504155064a55b5a716a520e1f2ac" I0918 20:38:39.734030 22374 raft_consensus.cc:2890] T ba7b5078d5404513ad4b6aa39b249244 P 2f1a504155064a55b5a716a520e1f2ac [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:39.731567 22434 raft_consensus.cc:472] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:39.745627 22373 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ba7b5078d5404513ad4b6aa39b249244" candidate_uuid: "1197af9577844861bc2218a92e37c173" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f1a504155064a55b5a716a520e1f2ac" is_pre_election: true I0918 20:38:39.754653 22434 raft_consensus.cc:2890] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:39.757242 22374 raft_consensus.cc:2350] T ba7b5078d5404513ad4b6aa39b249244 P 2f1a504155064a55b5a716a520e1f2ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bb3d9a778a4d8e9a3987f0a6d863b0 in term 1. I0918 20:38:39.757894 22373 raft_consensus.cc:2325] T ba7b5078d5404513ad4b6aa39b249244 P 2f1a504155064a55b5a716a520e1f2ac [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1197af9577844861bc2218a92e37c173 for term 1 because replica is already servicing an update from a current leader or another vote. I0918 20:38:39.758875 22441 raft_consensus.cc:494] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } I0918 20:38:39.760650 22302 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ba7b5078d5404513ad4b6aa39b249244" candidate_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1197af9577844861bc2218a92e37c173" I0918 20:38:39.762576 22302 raft_consensus.cc:2890] T ba7b5078d5404513ad4b6aa39b249244 P 1197af9577844861bc2218a92e37c173 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:39.760828 22227 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ba7b5078d5404513ad4b6aa39b249244" candidate_uuid: "1197af9577844861bc2218a92e37c173" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" is_pre_election: true I0918 20:38:39.764221 22227 raft_consensus.cc:2280] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1197af9577844861bc2218a92e37c173 in current term 1: Already voted for candidate b4bb3d9a778a4d8e9a3987f0a6d863b0 in this term. I0918 20:38:39.769922 22302 raft_consensus.cc:2350] T ba7b5078d5404513ad4b6aa39b249244 P 1197af9577844861bc2218a92e37c173 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bb3d9a778a4d8e9a3987f0a6d863b0 in term 1. I0918 20:38:39.761552 22184 leader_election.cc:310] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1a504155064a55b5a716a520e1f2ac, b4bb3d9a778a4d8e9a3987f0a6d863b0; no voters: I0918 20:38:39.772433 22447 raft_consensus.cc:2644] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:39.772222 22434 raft_consensus.cc:494] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } I0918 20:38:39.773164 22302 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "517a906cb7664795939d3758fc7d7a17" candidate_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1197af9577844861bc2218a92e37c173" I0918 20:38:39.773970 22434 leader_election.cc:296] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [CANDIDATE]: Term 1 election: Requested vote from peers 1197af9577844861bc2218a92e37c173 (127.18.222.2:35363), 2f1a504155064a55b5a716a520e1f2ac (127.18.222.3:46347) I0918 20:38:39.775135 22374 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "517a906cb7664795939d3758fc7d7a17" candidate_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f1a504155064a55b5a716a520e1f2ac" I0918 20:38:39.771667 22441 leader_election.cc:296] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [CANDIDATE]: Term 1 election: Requested vote from peers 1197af9577844861bc2218a92e37c173 (127.18.222.2:35363), 2f1a504155064a55b5a716a520e1f2ac (127.18.222.3:46347) I0918 20:38:39.776248 22374 raft_consensus.cc:2890] T 517a906cb7664795939d3758fc7d7a17 P 2f1a504155064a55b5a716a520e1f2ac [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:39.777302 22302 raft_consensus.cc:2890] T 517a906cb7664795939d3758fc7d7a17 P 1197af9577844861bc2218a92e37c173 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:39.779364 22373 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bd65e74409ad46799a393e20fbc31c54" candidate_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f1a504155064a55b5a716a520e1f2ac" I0918 20:38:39.780047 22373 raft_consensus.cc:2890] T bd65e74409ad46799a393e20fbc31c54 P 2f1a504155064a55b5a716a520e1f2ac [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:39.783406 22266 leader_election.cc:310] T ba7b5078d5404513ad4b6aa39b249244 P 1197af9577844861bc2218a92e37c173 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1197af9577844861bc2218a92e37c173; no voters: 2f1a504155064a55b5a716a520e1f2ac, b4bb3d9a778a4d8e9a3987f0a6d863b0 I0918 20:38:39.783998 22301 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bd65e74409ad46799a393e20fbc31c54" candidate_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1197af9577844861bc2218a92e37c173" I0918 20:38:39.784662 22301 raft_consensus.cc:2890] T bd65e74409ad46799a393e20fbc31c54 P 1197af9577844861bc2218a92e37c173 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:39.784809 22429 raft_consensus.cc:2592] T ba7b5078d5404513ad4b6aa39b249244 P 1197af9577844861bc2218a92e37c173 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0918 20:38:39.787605 22373 raft_consensus.cc:2350] T bd65e74409ad46799a393e20fbc31c54 P 2f1a504155064a55b5a716a520e1f2ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bb3d9a778a4d8e9a3987f0a6d863b0 in term 1. I0918 20:38:39.789463 22184 leader_election.cc:310] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1a504155064a55b5a716a520e1f2ac, b4bb3d9a778a4d8e9a3987f0a6d863b0; no voters: I0918 20:38:39.790496 22441 raft_consensus.cc:2644] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:39.795398 22374 raft_consensus.cc:2350] T 517a906cb7664795939d3758fc7d7a17 P 2f1a504155064a55b5a716a520e1f2ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bb3d9a778a4d8e9a3987f0a6d863b0 in term 1. I0918 20:38:39.790992 22301 raft_consensus.cc:2350] T bd65e74409ad46799a393e20fbc31c54 P 1197af9577844861bc2218a92e37c173 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bb3d9a778a4d8e9a3987f0a6d863b0 in term 1. I0918 20:38:39.796809 22184 leader_election.cc:310] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1a504155064a55b5a716a520e1f2ac, b4bb3d9a778a4d8e9a3987f0a6d863b0; no voters: I0918 20:38:39.801151 22447 raft_consensus.cc:667] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 1 LEADER]: Becoming Leader. State: Replica: b4bb3d9a778a4d8e9a3987f0a6d863b0, State: Running, Role: LEADER I0918 20:38:39.802814 22434 raft_consensus.cc:2644] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:39.803237 22434 raft_consensus.cc:667] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 1 LEADER]: Becoming Leader. State: Replica: b4bb3d9a778a4d8e9a3987f0a6d863b0, State: Running, Role: LEADER I0918 20:38:39.792011 22441 raft_consensus.cc:667] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 1 LEADER]: Becoming Leader. State: Replica: b4bb3d9a778a4d8e9a3987f0a6d863b0, State: Running, Role: LEADER I0918 20:38:39.804471 22441 consensus_queue.cc:206] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } I0918 20:38:39.804509 22434 consensus_queue.cc:206] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } I0918 20:38:39.814841 22302 raft_consensus.cc:2350] T 517a906cb7664795939d3758fc7d7a17 P 1197af9577844861bc2218a92e37c173 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bb3d9a778a4d8e9a3987f0a6d863b0 in term 1. I0918 20:38:39.809728 22447 consensus_queue.cc:206] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } } peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } } I0918 20:38:39.832911 22076 catalog_manager.cc:4131] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 reported cstate change: term changed from 0 to 1, leader changed from to b4bb3d9a778a4d8e9a3987f0a6d863b0 (127.18.222.1). New cstate: current_term: 1 leader_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } health_report { overall_health: HEALTHY } } } I0918 20:38:39.887631 22076 catalog_manager.cc:4131] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 reported cstate change: term changed from 0 to 1, leader changed from to b4bb3d9a778a4d8e9a3987f0a6d863b0 (127.18.222.1). New cstate: current_term: 1 leader_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } health_report { overall_health: UNKNOWN } } } I0918 20:38:39.889181 22076 catalog_manager.cc:4131] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 reported cstate change: term changed from 0 to 1, leader changed from to b4bb3d9a778a4d8e9a3987f0a6d863b0 (127.18.222.1). New cstate: current_term: 1 leader_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bb3d9a778a4d8e9a3987f0a6d863b0" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 } health_report { overall_health: UNKNOWN } } } I0918 20:38:40.034759 19320 ksck_remote-test.cc:412] Generating row writes... I0918 20:38:40.249672 22447 consensus_queue.cc:984] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:38:40.257515 22447 consensus_queue.cc:984] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:38:40.287839 22434 consensus_queue.cc:984] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f1a504155064a55b5a716a520e1f2ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:38:40.287968 22441 consensus_queue.cc:984] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:38:40.330709 22447 consensus_queue.cc:984] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:40.447016 22434 consensus_queue.cc:984] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1197af9577844861bc2218a92e37c173" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 35363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s I0918 20:38:42.999425 22499 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45514 (ReqId={client: 502c9e9781f24a22a6b936c9746065c8, seq_no=0, attempt_no=0}) took 1820ms. Request Metrics: {"spinlock_wait_cycles":4992,"child_traces":[["txn",{"apply.queue_time_us":316,"num_ops":3240,"prepare.queue_time_us":21418,"prepare.run_cpu_time_us":294806,"prepare.run_wall_time_us":569520,"raft.queue_time_us":483,"raft.run_cpu_time_us":2624,"raft.run_wall_time_us":3919,"replication_time_us":125249,"spinlock_wait_cycles":3246848,"thread_start_us":24227,"threads_started":2,"wal-append.queue_time_us":3503}]]} W0918 20:38:43.202484 22407 outbound_call.cc:287] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.18.222.1:35785, user_credentials={real_user=slave}} blocked reactor thread for 200790us W0918 20:38:43.506139 22513 log.cc:669] Time spent T bd65e74409ad46799a393e20fbc31c54 P 1197af9577844861bc2218a92e37c173: Append to log took a long time: real 0.103s user 0.010s sys 0.000s W0918 20:38:43.605175 22510 log.cc:669] Time spent T bd65e74409ad46799a393e20fbc31c54 P 2f1a504155064a55b5a716a520e1f2ac: Append to log took a long time: real 0.092s user 0.007s sys 0.006s I0918 20:38:44.470445 22499 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45514 (ReqId={client: 502c9e9781f24a22a6b936c9746065c8, seq_no=3, attempt_no=0}) took 2276ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":1091,"mutex_wait_us":20410,"num_ops":2993,"prepare.queue_time_us":219,"prepare.run_cpu_time_us":342580,"prepare.run_wall_time_us":1118245,"raft.queue_time_us":26032,"raft.run_cpu_time_us":4074,"raft.run_wall_time_us":4431,"replication_time_us":324201,"spinlock_wait_cycles":518144}]]} W0918 20:38:44.557190 22407 outbound_call.cc:287] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.18.222.1:35785, user_credentials={real_user=slave}} blocked reactor thread for 82218us W0918 20:38:45.030048 22518 log.cc:669] Time spent T bd65e74409ad46799a393e20fbc31c54 P 2f1a504155064a55b5a716a520e1f2ac: Append to log took a long time: real 0.059s user 0.010s sys 0.000s W0918 20:38:45.115440 22181 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.222.2:35363, user_credentials={real_user=slave}} blocked reactor thread for 49147us I0918 20:38:46.494570 22522 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45514 (ReqId={client: 502c9e9781f24a22a6b936c9746065c8, seq_no=4, attempt_no=0}) took 2420ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":62149,"num_ops":2441,"prepare.queue_time_us":12445,"prepare.run_cpu_time_us":256901,"prepare.run_wall_time_us":850749,"raft.queue_time_us":234,"raft.run_cpu_time_us":1913,"raft.run_wall_time_us":1964,"replication_time_us":128915,"spinlock_wait_cycles":33358464,"thread_start_us":65216,"threads_started":2}]]} W0918 20:38:46.652489 22407 outbound_call.cc:287] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.18.222.1:35785, user_credentials={real_user=slave}} blocked reactor thread for 143971us W0918 20:38:46.653230 22184 outbound_call.cc:287] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.18.222.60:46633, user_credentials={real_user=slave}} blocked reactor thread for 44030.4us W0918 20:38:46.696681 22337 outbound_call.cc:287] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.18.222.62:46257, user_credentials={real_user=slave}} blocked reactor thread for 88136.8us I0918 20:38:47.179088 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:47.191475 22184 messenger.cc:381] Service unavailable: service kudu.tserver.TabletServerService not registered on TabletServer W0918 20:38:47.193848 22462 ksck_checksum.cc:285] failed to submit task to start additional tablet checksums: Service unavailable: The pool has been shut down. I0918 20:38:47.197517 22184 messenger.cc:381] Service unavailable: service kudu.tserver.TabletServerService not registered on TabletServer I0918 20:38:47.198076 22184 messenger.cc:381] Service unavailable: service kudu.tserver.TabletServerService not registered on TabletServer W0918 20:38:47.199340 22461 ksck_checksum.cc:285] failed to submit task to start additional tablet checksums: Service unavailable: The pool has been shut down. W0918 20:38:47.199914 22461 ksck_checksum.cc:285] failed to submit task to start additional tablet checksums: Service unavailable: The pool has been shut down. W0918 20:38:47.200363 22461 ksck_checksum.cc:285] failed to submit task to start additional tablet checksums: Service unavailable: The pool has been shut down. W0918 20:38:47.202308 22459 ksck_checksum.cc:285] failed to submit task to start additional tablet checksums: Service unavailable: The pool has been shut down. W0918 20:38:47.205116 22462 ksck_checksum.cc:285] failed to submit task to start additional tablet checksums: Service unavailable: The pool has been shut down. W0918 20:38:47.207284 22459 ksck_checksum.cc:285] failed to submit task to start additional tablet checksums: Service unavailable: The pool has been shut down. I0918 20:38:47.384600 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:47.385254 19320 tablet_replica.cc:273] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0: stopping tablet replica I0918 20:38:47.385869 19320 raft_consensus.cc:2147] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:47.387411 19320 raft_consensus.cc:2174] T 517a906cb7664795939d3758fc7d7a17 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:47.390625 19320 tablet_replica.cc:273] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0: stopping tablet replica I0918 20:38:47.391117 19320 raft_consensus.cc:2147] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:47.392009 19320 raft_consensus.cc:2174] T bd65e74409ad46799a393e20fbc31c54 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:47.461870 19320 tablet_replica.cc:273] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0: stopping tablet replica I0918 20:38:47.462621 19320 raft_consensus.cc:2147] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:47.463567 19320 raft_consensus.cc:2174] T ba7b5078d5404513ad4b6aa39b249244 P b4bb3d9a778a4d8e9a3987f0a6d863b0 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:47.495283 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:47.495615 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:47.496146 19320 tablet_server.cc:169] TabletServer shutdown complete. W0918 20:38:47.497325 22459 ksck_checksum.cc:285] failed to submit task to start additional tablet checksums: Service unavailable: The pool has been shut down. I0918 20:38:47.521414 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:47.589165 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:47.589712 19320 tablet_replica.cc:273] T bd65e74409ad46799a393e20fbc31c54 P 1197af9577844861bc2218a92e37c173: stopping tablet replica I0918 20:38:47.590162 19320 raft_consensus.cc:2147] T bd65e74409ad46799a393e20fbc31c54 P 1197af9577844861bc2218a92e37c173 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:47.590534 19320 raft_consensus.cc:2174] T bd65e74409ad46799a393e20fbc31c54 P 1197af9577844861bc2218a92e37c173 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:47.617624 19320 tablet_replica.cc:273] T 517a906cb7664795939d3758fc7d7a17 P 1197af9577844861bc2218a92e37c173: stopping tablet replica I0918 20:38:47.618202 19320 raft_consensus.cc:2147] T 517a906cb7664795939d3758fc7d7a17 P 1197af9577844861bc2218a92e37c173 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:47.618569 19320 raft_consensus.cc:2174] T 517a906cb7664795939d3758fc7d7a17 P 1197af9577844861bc2218a92e37c173 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:47.621424 19320 tablet_replica.cc:273] T ba7b5078d5404513ad4b6aa39b249244 P 1197af9577844861bc2218a92e37c173: stopping tablet replica I0918 20:38:47.621845 19320 raft_consensus.cc:2147] T ba7b5078d5404513ad4b6aa39b249244 P 1197af9577844861bc2218a92e37c173 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:47.622296 19320 raft_consensus.cc:2174] T ba7b5078d5404513ad4b6aa39b249244 P 1197af9577844861bc2218a92e37c173 [term 1 FOLLOWER]: Raft consensus is shut down! W0918 20:38:47.642063 22462 ksck_checksum.cc:285] failed to submit task to start additional tablet checksums: Service unavailable: The pool has been shut down. I0918 20:38:47.647851 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:47.648105 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:47.648428 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:47.663162 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:47.704197 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:47.704512 19320 tablet_replica.cc:273] T 517a906cb7664795939d3758fc7d7a17 P 2f1a504155064a55b5a716a520e1f2ac: stopping tablet replica I0918 20:38:47.704843 19320 raft_consensus.cc:2147] T 517a906cb7664795939d3758fc7d7a17 P 2f1a504155064a55b5a716a520e1f2ac [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:47.705106 19320 raft_consensus.cc:2174] T 517a906cb7664795939d3758fc7d7a17 P 2f1a504155064a55b5a716a520e1f2ac [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:47.707125 19320 tablet_replica.cc:273] T ba7b5078d5404513ad4b6aa39b249244 P 2f1a504155064a55b5a716a520e1f2ac: stopping tablet replica I0918 20:38:47.707414 19320 raft_consensus.cc:2147] T ba7b5078d5404513ad4b6aa39b249244 P 2f1a504155064a55b5a716a520e1f2ac [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:47.707639 19320 raft_consensus.cc:2174] T ba7b5078d5404513ad4b6aa39b249244 P 2f1a504155064a55b5a716a520e1f2ac [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:47.709501 19320 tablet_replica.cc:273] T bd65e74409ad46799a393e20fbc31c54 P 2f1a504155064a55b5a716a520e1f2ac: stopping tablet replica I0918 20:38:47.709769 19320 raft_consensus.cc:2147] T bd65e74409ad46799a393e20fbc31c54 P 2f1a504155064a55b5a716a520e1f2ac [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:47.710000 19320 raft_consensus.cc:2174] T bd65e74409ad46799a393e20fbc31c54 P 2f1a504155064a55b5a716a520e1f2ac [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:47.743659 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:47.743947 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:47.744313 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:47.756520 19320 master.cc:247] Master@127.18.222.62:46257 shutting down... I0918 20:38:47.781916 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:47.782320 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:47.782588 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P ccffb4cc11b349b4ae52a0a13eb59b94: stopping tablet replica I0918 20:38:47.800734 19320 master.cc:258] Master@127.18.222.62:46257 shutdown complete. I0918 20:38:47.809619 19320 master.cc:247] Master@127.18.222.61:41695 shutting down... I0918 20:38:47.831400 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:47.832232 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:47.832500 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 6feec5fc783e4905982503aeadbd8d16: stopping tablet replica I0918 20:38:47.851333 19320 master.cc:258] Master@127.18.222.61:41695 shutdown complete. I0918 20:38:47.862073 19320 master.cc:247] Master@127.18.222.60:46633 shutting down... I0918 20:38:47.885799 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:47.886142 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:47.886365 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P ab28e98d635942f098b2bcad0bfd652c: stopping tablet replica I0918 20:38:47.903937 19320 master.cc:258] Master@127.18.222.60:46633 shutdown complete. [ OK ] RemoteKsckTest.TestChecksumTimeout (10734 ms) [ RUN ] RemoteKsckTest.TestChecksumSnapshot I0918 20:38:47.938925 19320 test_util.cc:212] Using random seed: -756140205 I0918 20:38:47.939620 19320 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.18.222.62:38613,127.18.222.61:46291,127.18.222.60:33817 I0918 20:38:47.952069 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 141092us I0918 20:38:47.952415 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:47.952545 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:47.952752 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:47.952877 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:47.953832 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/master-0-root/instance: uuid: "50eed283ebb8415191b2a2ff60a43c25" format_stamp: "Formatted at 2019-09-18 20:38:47 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:47.963249 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.010s sys 0.001s I0918 20:38:47.964784 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:47.971359 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:47.971551 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/master-0-root uuid: "50eed283ebb8415191b2a2ff60a43c25" format_stamp: "Formatted at 2019-09-18 20:38:47 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:47.971880 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:48.040484 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:38613 I0918 20:38:48.042279 19320 webserver.cc:374] Webserver started at http://127.18.222.62:34165/ using document root and password file I0918 20:38:48.044060 22613 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:48.048784 22613 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:48.057931 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 141592us I0918 20:38:48.058547 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:48.058719 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:48.058938 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:48.059073 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:48.060204 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/master-1-root/instance: uuid: "1fe19deb62c543c3a2f16d29163424de" format_stamp: "Formatted at 2019-09-18 20:38:48 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:48.062113 22613 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } has no permanent_uuid. Determining permanent_uuid... W0918 20:38:48.063935 22613 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:46291: Network error: Client connection negotiation failed: client connection to 127.18.222.61:46291: connect: Connection refused (error 111) I0918 20:38:48.070716 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.007s sys 0.003s I0918 20:38:48.072490 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:48.080405 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:48.080624 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/master-1-root uuid: "1fe19deb62c543c3a2f16d29163424de" format_stamp: "Formatted at 2019-09-18 20:38:48 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:48.080883 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:48.099292 22613 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } attempt: 1 W0918 20:38:48.101299 22613 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:46291: Network error: Client connection negotiation failed: client connection to 127.18.222.61:46291: connect: Connection refused (error 111) I0918 20:38:48.133962 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:46291 I0918 20:38:48.136071 19320 webserver.cc:374] Webserver started at http://127.18.222.61:43521/ using document root and password file I0918 20:38:48.138325 22670 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:48.144838 22670 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:48.158643 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 141592us I0918 20:38:48.158807 22670 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:48.159123 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:48.159394 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:48.159672 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:48.159850 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:48.161023 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/master-2-root/instance: uuid: "b137a42405ad403182bd6e1b01fc43ec" format_stamp: "Formatted at 2019-09-18 20:38:48 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:48.162662 22613 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } attempt: 2 I0918 20:38:48.176420 22670 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:48.177165 22613 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:48.178434 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.017s user 0.016s sys 0.000s W0918 20:38:48.178586 22670 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:33817: Network error: Client connection negotiation failed: client connection to 127.18.222.60:33817: connect: Connection refused (error 111) W0918 20:38:48.179196 22613 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:33817: Network error: Client connection negotiation failed: client connection to 127.18.222.60:33817: connect: Connection refused (error 111) I0918 20:38:48.180603 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:48.188964 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I0918 20:38:48.189172 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/master-2-root uuid: "b137a42405ad403182bd6e1b01fc43ec" format_stamp: "Formatted at 2019-09-18 20:38:48 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:48.189448 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:48.232256 22670 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } attempt: 1 I0918 20:38:48.232540 22613 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } attempt: 1 W0918 20:38:48.234635 22613 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:33817: Network error: Client connection negotiation failed: client connection to 127.18.222.60:33817: connect: Connection refused (error 111) W0918 20:38:48.235453 22670 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:33817: Network error: Client connection negotiation failed: client connection to 127.18.222.60:33817: connect: Connection refused (error 111) I0918 20:38:48.275478 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:33817 I0918 20:38:48.277299 19320 webserver.cc:374] Webserver started at http://127.18.222.60:36961/ using document root and password file I0918 20:38:48.278240 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0918 20:38:48.279078 22728 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:48.284015 22728 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:48.293859 22670 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } attempt: 2 I0918 20:38:48.295141 22728 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:48.304184 22613 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } attempt: 2 I0918 20:38:48.309274 22728 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:48.316565 22670 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de: Bootstrap starting. I0918 20:38:48.320850 22670 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de: No blocks or log segments found. Creating new log. I0918 20:38:48.325755 22670 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de: No bootstrap required, opened a new log I0918 20:38:48.326683 22613 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25: Bootstrap starting. I0918 20:38:48.328121 22670 raft_consensus.cc:340] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } I0918 20:38:48.328896 22670 raft_consensus.cc:366] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:48.329123 22670 raft_consensus.cc:705] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe19deb62c543c3a2f16d29163424de, State: Initialized, Role: FOLLOWER I0918 20:38:48.329777 22728 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec: Bootstrap starting. I0918 20:38:48.329782 22613 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25: No blocks or log segments found. Creating new log. I0918 20:38:48.329771 22670 consensus_queue.cc:229] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } I0918 20:38:48.332163 22670 sys_catalog.cc:340] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } } I0918 20:38:48.333184 22670 sys_catalog.cc:343] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:48.332548 22733 sys_catalog.cc:340] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } } I0918 20:38:48.333887 22733 sys_catalog.cc:343] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:48.334237 22670 sys_catalog.cc:428] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de [sys.catalog]: configured and running, proceeding with master startup. I0918 20:38:48.334859 22613 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25: No bootstrap required, opened a new log I0918 20:38:48.336464 22728 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec: No blocks or log segments found. Creating new log. I0918 20:38:48.337514 22613 raft_consensus.cc:340] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } I0918 20:38:48.338289 22613 raft_consensus.cc:366] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely W0918 20:38:48.338387 22735 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de: acquiring CA information for follower catalog manager: Not found: root CA entry not found I0918 20:38:48.338548 22613 raft_consensus.cc:705] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50eed283ebb8415191b2a2ff60a43c25, State: Initialized, Role: FOLLOWER W0918 20:38:48.338692 22735 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:48.339277 22613 consensus_queue.cc:229] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } I0918 20:38:48.340766 22728 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec: No bootstrap required, opened a new log I0918 20:38:48.341572 22737 sys_catalog.cc:340] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } } I0918 20:38:48.341434 22613 sys_catalog.cc:340] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } } I0918 20:38:48.342142 22737 sys_catalog.cc:343] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:48.342263 22613 sys_catalog.cc:343] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:48.343516 22613 sys_catalog.cc:428] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:38:48.343410 22728 raft_consensus.cc:340] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } I0918 20:38:48.344313 22728 raft_consensus.cc:366] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:48.344557 22728 raft_consensus.cc:705] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b137a42405ad403182bd6e1b01fc43ec, State: Initialized, Role: FOLLOWER I0918 20:38:48.344880 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 I0918 20:38:48.345156 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 I0918 20:38:48.345408 22728 consensus_queue.cc:229] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } I0918 20:38:48.347064 22728 sys_catalog.cc:340] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } } W0918 20:38:48.347497 22738 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25: acquiring CA information for follower catalog manager: Not found: root CA entry not found I0918 20:38:48.347574 22728 sys_catalog.cc:343] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [sys.catalog]: This master's current role is: FOLLOWER W0918 20:38:48.348057 22738 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:48.347415 22739 sys_catalog.cc:340] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } } I0918 20:38:48.348402 22739 sys_catalog.cc:343] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:48.348966 22728 sys_catalog.cc:428] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [sys.catalog]: configured and running, proceeding with master startup. W0918 20:38:48.349277 22626 tablet.cc:1763] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de: Can't schedule compaction. Clean time has not been advanced past its initial value. W0918 20:38:48.352773 22740 catalog_manager.cc:1155] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:38:48.353073 22740 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:48.366722 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 141592us I0918 20:38:48.367094 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:48.367239 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:48.367427 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:48.367539 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:48.368515 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/ts-0-root/instance: uuid: "06662902cf85459a86d2fc0759450bcf" format_stamp: "Formatted at 2019-09-18 20:38:48 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:48.378144 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.010s sys 0.000s I0918 20:38:48.379607 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:48.387820 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:38:48.388025 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/ts-0-root uuid: "06662902cf85459a86d2fc0759450bcf" format_stamp: "Formatted at 2019-09-18 20:38:48 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:48.388255 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:48.388972 22739 raft_consensus.cc:472] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:48.389516 22739 raft_consensus.cc:494] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } I0918 20:38:48.391569 22739 leader_election.cc:296] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50eed283ebb8415191b2a2ff60a43c25 (127.18.222.62:38613), 1fe19deb62c543c3a2f16d29163424de (127.18.222.61:46291) I0918 20:38:48.392788 22647 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b137a42405ad403182bd6e1b01fc43ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe19deb62c543c3a2f16d29163424de" is_pre_election: true I0918 20:38:48.392704 22590 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b137a42405ad403182bd6e1b01fc43ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50eed283ebb8415191b2a2ff60a43c25" is_pre_election: true I0918 20:38:48.393518 22647 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b137a42405ad403182bd6e1b01fc43ec in term 0. I0918 20:38:48.394124 22590 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b137a42405ad403182bd6e1b01fc43ec in term 0. I0918 20:38:48.394990 22677 leader_election.cc:310] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe19deb62c543c3a2f16d29163424de, b137a42405ad403182bd6e1b01fc43ec; no voters: I0918 20:38:48.395915 22739 raft_consensus.cc:2644] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:48.396205 22739 raft_consensus.cc:472] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:48.396446 22739 raft_consensus.cc:2890] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:48.400482 22739 raft_consensus.cc:494] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } I0918 20:38:48.401760 22739 leader_election.cc:296] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [CANDIDATE]: Term 1 election: Requested vote from peers 50eed283ebb8415191b2a2ff60a43c25 (127.18.222.62:38613), 1fe19deb62c543c3a2f16d29163424de (127.18.222.61:46291) I0918 20:38:48.402774 22647 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b137a42405ad403182bd6e1b01fc43ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe19deb62c543c3a2f16d29163424de" I0918 20:38:48.402731 22590 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b137a42405ad403182bd6e1b01fc43ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50eed283ebb8415191b2a2ff60a43c25" I0918 20:38:48.403558 22647 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:48.403586 22590 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:48.409723 22590 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b137a42405ad403182bd6e1b01fc43ec in term 1. I0918 20:38:48.409723 22647 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b137a42405ad403182bd6e1b01fc43ec in term 1. I0918 20:38:48.411039 22678 leader_election.cc:310] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50eed283ebb8415191b2a2ff60a43c25, b137a42405ad403182bd6e1b01fc43ec; no voters: I0918 20:38:48.411866 22739 raft_consensus.cc:2644] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:48.413167 22739 raft_consensus.cc:667] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [term 1 LEADER]: Becoming Leader. State: Replica: b137a42405ad403182bd6e1b01fc43ec, State: Running, Role: LEADER I0918 20:38:48.413727 22739 consensus_queue.cc:206] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } I0918 20:38:48.417912 22747 sys_catalog.cc:340] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader b137a42405ad403182bd6e1b01fc43ec. Latest consensus state: current_term: 1 leader_uuid: "b137a42405ad403182bd6e1b01fc43ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } } I0918 20:38:48.418464 22747 sys_catalog.cc:343] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [sys.catalog]: This master's current role is: LEADER I0918 20:38:48.420066 22749 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:38:48.422551 22749 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:38:48.434478 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:48.434687 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:48.443017 22647 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de [term 1 FOLLOWER]: Refusing update from remote peer b137a42405ad403182bd6e1b01fc43ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:48.443210 22590 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25 [term 1 FOLLOWER]: Refusing update from remote peer b137a42405ad403182bd6e1b01fc43ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:48.444386 22747 consensus_queue.cc:984] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:48.445212 22739 consensus_queue.cc:984] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:48.458469 22733 sys_catalog.cc:340] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de [sys.catalog]: SysCatalogTable state changed. Reason: New leader b137a42405ad403182bd6e1b01fc43ec. Latest consensus state: current_term: 1 leader_uuid: "b137a42405ad403182bd6e1b01fc43ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } } I0918 20:38:48.458834 22737 sys_catalog.cc:340] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b137a42405ad403182bd6e1b01fc43ec. Latest consensus state: current_term: 1 leader_uuid: "b137a42405ad403182bd6e1b01fc43ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } } I0918 20:38:48.459760 22733 sys_catalog.cc:343] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:48.459986 22737 sys_catalog.cc:343] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:48.470173 22739 sys_catalog.cc:340] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b137a42405ad403182bd6e1b01fc43ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } } I0918 20:38:48.470744 22733 sys_catalog.cc:340] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b137a42405ad403182bd6e1b01fc43ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } } I0918 20:38:48.471480 22733 sys_catalog.cc:343] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:48.471401 22739 sys_catalog.cc:343] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [sys.catalog]: This master's current role is: LEADER I0918 20:38:48.469010 22747 sys_catalog.cc:340] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b137a42405ad403182bd6e1b01fc43ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } } I0918 20:38:48.473639 22747 sys_catalog.cc:343] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [sys.catalog]: This master's current role is: LEADER I0918 20:38:48.474453 22749 catalog_manager.cc:981] Generated new certificate authority record I0918 20:38:48.476511 22749 catalog_manager.cc:1106] Loading token signing keys... I0918 20:38:48.485534 22737 sys_catalog.cc:340] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b137a42405ad403182bd6e1b01fc43ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed283ebb8415191b2a2ff60a43c25" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38613 } } peers { permanent_uuid: "1fe19deb62c543c3a2f16d29163424de" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 46291 } } peers { permanent_uuid: "b137a42405ad403182bd6e1b01fc43ec" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33817 } } } I0918 20:38:48.486416 22737 sys_catalog.cc:343] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:48.499219 22749 catalog_manager.cc:4446] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec: Generated new TSK 0 I0918 20:38:48.500295 22749 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:38:48.525023 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:35617 I0918 20:38:48.526872 19320 webserver.cc:374] Webserver started at http://127.18.222.1:40995/ using document root and password file I0918 20:38:48.556042 22817 heartbeater.cc:322] Connected to a master server at 127.18.222.60:33817 I0918 20:38:48.556429 22817 heartbeater.cc:406] Registering TS with master... I0918 20:38:48.557013 22817 heartbeater.cc:455] Master 127.18.222.60:33817 requested a full tablet report, sending... I0918 20:38:48.558528 22818 heartbeater.cc:322] Connected to a master server at 127.18.222.61:46291 I0918 20:38:48.558990 22818 heartbeater.cc:406] Registering TS with master... I0918 20:38:48.559123 22695 ts_manager.cc:186] Registered new tserver with Master: 06662902cf85459a86d2fc0759450bcf (127.18.222.1:35617) I0918 20:38:48.559592 22818 heartbeater.cc:455] Master 127.18.222.61:46291 requested a full tablet report, sending... I0918 20:38:48.561867 22695 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46850 I0918 20:38:48.562266 22637 ts_manager.cc:186] Registered new tserver with Master: 06662902cf85459a86d2fc0759450bcf (127.18.222.1:35617) I0918 20:38:48.565335 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 141592us I0918 20:38:48.565495 22819 heartbeater.cc:322] Connected to a master server at 127.18.222.62:38613 I0918 20:38:48.565845 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:48.565868 22819 heartbeater.cc:406] Registering TS with master... I0918 20:38:48.566251 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:48.566578 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:48.566594 22819 heartbeater.cc:455] Master 127.18.222.62:38613 requested a full tablet report, sending... I0918 20:38:48.566869 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:48.568238 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/ts-1-root/instance: uuid: "910125a69a974cec9efdf2fb67e7f9d5" format_stamp: "Formatted at 2019-09-18 20:38:48 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:48.569118 22580 ts_manager.cc:186] Registered new tserver with Master: 06662902cf85459a86d2fc0759450bcf (127.18.222.1:35617) I0918 20:38:48.575989 22817 heartbeater.cc:447] Master 127.18.222.60:33817 was elected leader, sending a full tablet report... I0918 20:38:48.580070 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.006s sys 0.005s I0918 20:38:48.582041 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:48.591609 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:48.591882 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/ts-1-root uuid: "910125a69a974cec9efdf2fb67e7f9d5" format_stamp: "Formatted at 2019-09-18 20:38:48 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:48.592149 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:48.634227 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:48.634510 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:48.680723 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.2:40963 I0918 20:38:48.682665 19320 webserver.cc:374] Webserver started at http://127.18.222.2:36995/ using document root and password file I0918 20:38:48.719439 22890 heartbeater.cc:322] Connected to a master server at 127.18.222.61:46291 I0918 20:38:48.719930 22890 heartbeater.cc:406] Registering TS with master... I0918 20:38:48.720631 22890 heartbeater.cc:455] Master 127.18.222.61:46291 requested a full tablet report, sending... I0918 20:38:48.723276 22637 ts_manager.cc:186] Registered new tserver with Master: 910125a69a974cec9efdf2fb67e7f9d5 (127.18.222.2:40963) I0918 20:38:48.726536 22891 heartbeater.cc:322] Connected to a master server at 127.18.222.62:38613 I0918 20:38:48.726837 22891 heartbeater.cc:406] Registering TS with master... I0918 20:38:48.727438 22891 heartbeater.cc:455] Master 127.18.222.62:38613 requested a full tablet report, sending... I0918 20:38:48.729962 22580 ts_manager.cc:186] Registered new tserver with Master: 910125a69a974cec9efdf2fb67e7f9d5 (127.18.222.2:40963) I0918 20:38:48.730804 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 141592us I0918 20:38:48.731218 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:48.731400 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:48.731428 22888 heartbeater.cc:322] Connected to a master server at 127.18.222.60:33817 I0918 20:38:48.731621 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:48.731865 22888 heartbeater.cc:406] Registering TS with master... I0918 20:38:48.731889 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:48.732399 22888 heartbeater.cc:455] Master 127.18.222.60:33817 requested a full tablet report, sending... I0918 20:38:48.733111 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/ts-2-root/instance: uuid: "244f0d3980ae451aac40562d8b441de0" format_stamp: "Formatted at 2019-09-18 20:38:48 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:48.733989 22695 ts_manager.cc:186] Registered new tserver with Master: 910125a69a974cec9efdf2fb67e7f9d5 (127.18.222.2:40963) I0918 20:38:48.735401 22695 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46856 I0918 20:38:48.747225 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.014s user 0.010s sys 0.004s I0918 20:38:48.748483 22888 heartbeater.cc:447] Master 127.18.222.60:33817 was elected leader, sending a full tablet report... I0918 20:38:48.749344 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:48.761502 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:48.761732 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/ts-2-root uuid: "244f0d3980ae451aac40562d8b441de0" format_stamp: "Formatted at 2019-09-18 20:38:48 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:48.762003 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1568839089958813-19320/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:48.797312 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:48.797526 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:48.874033 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.3:43223 I0918 20:38:48.876325 19320 webserver.cc:374] Webserver started at http://127.18.222.3:34831/ using document root and password file I0918 20:38:48.944195 22959 heartbeater.cc:322] Connected to a master server at 127.18.222.60:33817 I0918 20:38:48.944566 22959 heartbeater.cc:406] Registering TS with master... I0918 20:38:48.945185 22960 heartbeater.cc:322] Connected to a master server at 127.18.222.61:46291 I0918 20:38:48.945147 22959 heartbeater.cc:455] Master 127.18.222.60:33817 requested a full tablet report, sending... I0918 20:38:48.945551 22960 heartbeater.cc:406] Registering TS with master... I0918 20:38:48.946132 22960 heartbeater.cc:455] Master 127.18.222.61:46291 requested a full tablet report, sending... I0918 20:38:48.948575 22636 ts_manager.cc:186] Registered new tserver with Master: 244f0d3980ae451aac40562d8b441de0 (127.18.222.3:43223) I0918 20:38:48.950099 22695 ts_manager.cc:186] Registered new tserver with Master: 244f0d3980ae451aac40562d8b441de0 (127.18.222.3:43223) I0918 20:38:48.952625 22695 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46862 I0918 20:38:48.962222 22961 heartbeater.cc:322] Connected to a master server at 127.18.222.62:38613 I0918 20:38:48.962576 22961 heartbeater.cc:406] Registering TS with master... I0918 20:38:48.963007 22961 heartbeater.cc:455] Master 127.18.222.62:38613 requested a full tablet report, sending... I0918 20:38:48.965791 22959 heartbeater.cc:447] Master 127.18.222.60:33817 was elected leader, sending a full tablet report... I0918 20:38:48.965845 22579 ts_manager.cc:186] Registered new tserver with Master: 244f0d3980ae451aac40562d8b441de0 (127.18.222.3:43223) I0918 20:38:48.967109 19320 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.031297341s I0918 20:38:49.045136 22694 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46872: name: "ksck-test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\n\000\000\000\004\001\024\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:38:49.047096 22694 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:38:49.182541 22784 tablet_service.cc:1022] Processing CreateTablet for tablet b258257413fa48e68e29ef2a9ee9e145 (table=ksck-test-table [id=e65849c6b17d41c7bbc273584947993b]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:49.183672 22784 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet b258257413fa48e68e29ef2a9ee9e145. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:49.184988 22783 tablet_service.cc:1022] Processing CreateTablet for tablet 4ecd01ad0981467ba83c8134e92befc0 (table=ksck-test-table [id=e65849c6b17d41c7bbc273584947993b]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:49.186024 22783 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 4ecd01ad0981467ba83c8134e92befc0. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:49.199846 22782 tablet_service.cc:1022] Processing CreateTablet for tablet 5175d7454e8a47848c04989b12e67f49 (table=ksck-test-table [id=e65849c6b17d41c7bbc273584947993b]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:49.200996 22782 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 5175d7454e8a47848c04989b12e67f49. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:49.201232 22855 tablet_service.cc:1022] Processing CreateTablet for tablet b258257413fa48e68e29ef2a9ee9e145 (table=ksck-test-table [id=e65849c6b17d41c7bbc273584947993b]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:49.204725 22853 tablet_service.cc:1022] Processing CreateTablet for tablet 5175d7454e8a47848c04989b12e67f49 (table=ksck-test-table [id=e65849c6b17d41c7bbc273584947993b]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:49.205787 22854 tablet_service.cc:1022] Processing CreateTablet for tablet 4ecd01ad0981467ba83c8134e92befc0 (table=ksck-test-table [id=e65849c6b17d41c7bbc273584947993b]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:49.206322 22926 tablet_service.cc:1022] Processing CreateTablet for tablet b258257413fa48e68e29ef2a9ee9e145 (table=ksck-test-table [id=e65849c6b17d41c7bbc273584947993b]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:49.206854 22855 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet b258257413fa48e68e29ef2a9ee9e145. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:49.207252 22926 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet b258257413fa48e68e29ef2a9ee9e145. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:49.208006 22853 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 5175d7454e8a47848c04989b12e67f49. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:49.212050 22854 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 4ecd01ad0981467ba83c8134e92befc0. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:49.216984 22925 tablet_service.cc:1022] Processing CreateTablet for tablet 4ecd01ad0981467ba83c8134e92befc0 (table=ksck-test-table [id=e65849c6b17d41c7bbc273584947993b]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:49.218075 22925 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 4ecd01ad0981467ba83c8134e92befc0. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:49.221632 22924 tablet_service.cc:1022] Processing CreateTablet for tablet 5175d7454e8a47848c04989b12e67f49 (table=ksck-test-table [id=e65849c6b17d41c7bbc273584947993b]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:49.222678 22924 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 5175d7454e8a47848c04989b12e67f49. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:49.226181 22988 tablet_bootstrap.cc:439] T b258257413fa48e68e29ef2a9ee9e145 P 06662902cf85459a86d2fc0759450bcf: Bootstrap starting. I0918 20:38:49.266794 22989 tablet_bootstrap.cc:439] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5: Bootstrap starting. I0918 20:38:49.284546 22988 tablet_bootstrap.cc:590] T b258257413fa48e68e29ef2a9ee9e145 P 06662902cf85459a86d2fc0759450bcf: No blocks or log segments found. Creating new log. I0918 20:38:49.285639 22990 tablet_bootstrap.cc:439] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0: Bootstrap starting. I0918 20:38:49.291545 22990 tablet_bootstrap.cc:590] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0: No blocks or log segments found. Creating new log. I0918 20:38:49.294250 22989 tablet_bootstrap.cc:590] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5: No blocks or log segments found. Creating new log. I0918 20:38:49.299959 22990 tablet_bootstrap.cc:439] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0: No bootstrap required, opened a new log I0918 20:38:49.300550 22990 ts_tablet_manager.cc:1085] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s I0918 20:38:49.304117 22990 raft_consensus.cc:340] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } I0918 20:38:49.306118 22990 raft_consensus.cc:366] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:49.306414 22990 raft_consensus.cc:705] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244f0d3980ae451aac40562d8b441de0, State: Initialized, Role: FOLLOWER I0918 20:38:49.307063 22990 consensus_queue.cc:229] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } I0918 20:38:49.307657 22988 tablet_bootstrap.cc:439] T b258257413fa48e68e29ef2a9ee9e145 P 06662902cf85459a86d2fc0759450bcf: No bootstrap required, opened a new log I0918 20:38:49.308567 22988 ts_tablet_manager.cc:1085] T b258257413fa48e68e29ef2a9ee9e145 P 06662902cf85459a86d2fc0759450bcf: Time spent bootstrapping tablet: real 0.083s user 0.007s sys 0.000s I0918 20:38:49.310421 22989 tablet_bootstrap.cc:439] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5: No bootstrap required, opened a new log I0918 20:38:49.310972 22989 ts_tablet_manager.cc:1085] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.003s I0918 20:38:49.313949 22989 raft_consensus.cc:340] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } I0918 20:38:49.314826 22989 raft_consensus.cc:366] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:49.315148 22989 raft_consensus.cc:705] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910125a69a974cec9efdf2fb67e7f9d5, State: Initialized, Role: FOLLOWER I0918 20:38:49.316256 22989 consensus_queue.cc:229] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } I0918 20:38:49.318912 22988 raft_consensus.cc:340] T b258257413fa48e68e29ef2a9ee9e145 P 06662902cf85459a86d2fc0759450bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } I0918 20:38:49.319816 22988 raft_consensus.cc:366] T b258257413fa48e68e29ef2a9ee9e145 P 06662902cf85459a86d2fc0759450bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:49.320099 22988 raft_consensus.cc:705] T b258257413fa48e68e29ef2a9ee9e145 P 06662902cf85459a86d2fc0759450bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06662902cf85459a86d2fc0759450bcf, State: Initialized, Role: FOLLOWER I0918 20:38:49.320886 22988 consensus_queue.cc:229] T b258257413fa48e68e29ef2a9ee9e145 P 06662902cf85459a86d2fc0759450bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } I0918 20:38:49.336608 22990 ts_tablet_manager.cc:1113] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0: Time spent starting tablet: real 0.036s user 0.008s sys 0.011s I0918 20:38:49.346602 22994 raft_consensus.cc:472] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:49.349261 22994 raft_consensus.cc:494] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } I0918 20:38:49.358611 22735 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de: acquiring CA information for follower catalog manager: success I0918 20:38:49.361572 22990 tablet_bootstrap.cc:439] T 4ecd01ad0981467ba83c8134e92befc0 P 244f0d3980ae451aac40562d8b441de0: Bootstrap starting. I0918 20:38:49.373745 22989 ts_tablet_manager.cc:1113] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5: Time spent starting tablet: real 0.063s user 0.017s sys 0.016s I0918 20:38:49.374704 22989 tablet_bootstrap.cc:439] T 5175d7454e8a47848c04989b12e67f49 P 910125a69a974cec9efdf2fb67e7f9d5: Bootstrap starting. I0918 20:38:49.383409 22988 ts_tablet_manager.cc:1113] T b258257413fa48e68e29ef2a9ee9e145 P 06662902cf85459a86d2fc0759450bcf: Time spent starting tablet: real 0.075s user 0.018s sys 0.011s I0918 20:38:49.391036 22988 tablet_bootstrap.cc:439] T 4ecd01ad0981467ba83c8134e92befc0 P 06662902cf85459a86d2fc0759450bcf: Bootstrap starting. I0918 20:38:49.395104 22988 tablet_bootstrap.cc:590] T 4ecd01ad0981467ba83c8134e92befc0 P 06662902cf85459a86d2fc0759450bcf: No blocks or log segments found. Creating new log. I0918 20:38:49.385938 22994 leader_election.cc:296] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06662902cf85459a86d2fc0759450bcf (127.18.222.1:35617), 910125a69a974cec9efdf2fb67e7f9d5 (127.18.222.2:40963) I0918 20:38:49.410858 22794 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "b258257413fa48e68e29ef2a9ee9e145" candidate_uuid: "244f0d3980ae451aac40562d8b441de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06662902cf85459a86d2fc0759450bcf" is_pre_election: true I0918 20:38:49.411620 22794 raft_consensus.cc:2350] T b258257413fa48e68e29ef2a9ee9e145 P 06662902cf85459a86d2fc0759450bcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244f0d3980ae451aac40562d8b441de0 in term 0. I0918 20:38:49.413632 22897 leader_election.cc:310] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06662902cf85459a86d2fc0759450bcf, 244f0d3980ae451aac40562d8b441de0; no voters: I0918 20:38:49.414758 22994 raft_consensus.cc:2644] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:49.415194 22994 raft_consensus.cc:472] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:49.416960 22994 raft_consensus.cc:2890] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:49.420440 22990 tablet_bootstrap.cc:590] T 4ecd01ad0981467ba83c8134e92befc0 P 244f0d3980ae451aac40562d8b441de0: No blocks or log segments found. Creating new log. I0918 20:38:49.422471 22865 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "b258257413fa48e68e29ef2a9ee9e145" candidate_uuid: "244f0d3980ae451aac40562d8b441de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "910125a69a974cec9efdf2fb67e7f9d5" is_pre_election: true I0918 20:38:49.423944 22735 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 W0918 20:38:49.424355 22900 leader_election.cc:348] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 910125a69a974cec9efdf2fb67e7f9d5 (127.18.222.2:40963): Illegal state: must be running to vote when last-logged opid is not known I0918 20:38:49.422823 22738 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25: acquiring CA information for follower catalog manager: success I0918 20:38:49.429078 22738 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:38:49.435272 22990 tablet_bootstrap.cc:439] T 4ecd01ad0981467ba83c8134e92befc0 P 244f0d3980ae451aac40562d8b441de0: No bootstrap required, opened a new log I0918 20:38:49.436038 22990 ts_tablet_manager.cc:1085] T 4ecd01ad0981467ba83c8134e92befc0 P 244f0d3980ae451aac40562d8b441de0: Time spent bootstrapping tablet: real 0.096s user 0.003s sys 0.007s I0918 20:38:49.437206 22988 tablet_bootstrap.cc:439] T 4ecd01ad0981467ba83c8134e92befc0 P 06662902cf85459a86d2fc0759450bcf: No bootstrap required, opened a new log I0918 20:38:49.439129 22988 ts_tablet_manager.cc:1085] T 4ecd01ad0981467ba83c8134e92befc0 P 06662902cf85459a86d2fc0759450bcf: Time spent bootstrapping tablet: real 0.048s user 0.016s sys 0.005s I0918 20:38:49.441015 22988 raft_consensus.cc:340] T 4ecd01ad0981467ba83c8134e92befc0 P 06662902cf85459a86d2fc0759450bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } I0918 20:38:49.452380 22988 raft_consensus.cc:366] T 4ecd01ad0981467ba83c8134e92befc0 P 06662902cf85459a86d2fc0759450bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:49.452759 22988 raft_consensus.cc:705] T 4ecd01ad0981467ba83c8134e92befc0 P 06662902cf85459a86d2fc0759450bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06662902cf85459a86d2fc0759450bcf, State: Initialized, Role: FOLLOWER I0918 20:38:49.438282 22990 raft_consensus.cc:340] T 4ecd01ad0981467ba83c8134e92befc0 P 244f0d3980ae451aac40562d8b441de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } I0918 20:38:49.453145 22990 raft_consensus.cc:366] T 4ecd01ad0981467ba83c8134e92befc0 P 244f0d3980ae451aac40562d8b441de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:49.453406 22990 raft_consensus.cc:705] T 4ecd01ad0981467ba83c8134e92befc0 P 244f0d3980ae451aac40562d8b441de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244f0d3980ae451aac40562d8b441de0, State: Initialized, Role: FOLLOWER I0918 20:38:49.454210 22990 consensus_queue.cc:229] T 4ecd01ad0981467ba83c8134e92befc0 P 244f0d3980ae451aac40562d8b441de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } I0918 20:38:49.451452 22989 tablet_bootstrap.cc:590] T 5175d7454e8a47848c04989b12e67f49 P 910125a69a974cec9efdf2fb67e7f9d5: No blocks or log segments found. Creating new log. I0918 20:38:49.465636 22990 ts_tablet_manager.cc:1113] T 4ecd01ad0981467ba83c8134e92befc0 P 244f0d3980ae451aac40562d8b441de0: Time spent starting tablet: real 0.029s user 0.003s sys 0.002s I0918 20:38:49.466485 22990 tablet_bootstrap.cc:439] T 5175d7454e8a47848c04989b12e67f49 P 244f0d3980ae451aac40562d8b441de0: Bootstrap starting. I0918 20:38:49.480268 22988 consensus_queue.cc:229] T 4ecd01ad0981467ba83c8134e92befc0 P 06662902cf85459a86d2fc0759450bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } I0918 20:38:49.481411 22994 raft_consensus.cc:494] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } I0918 20:38:49.484875 22989 tablet_bootstrap.cc:439] T 5175d7454e8a47848c04989b12e67f49 P 910125a69a974cec9efdf2fb67e7f9d5: No bootstrap required, opened a new log I0918 20:38:49.485551 22989 ts_tablet_manager.cc:1085] T 5175d7454e8a47848c04989b12e67f49 P 910125a69a974cec9efdf2fb67e7f9d5: Time spent bootstrapping tablet: real 0.111s user 0.006s sys 0.002s I0918 20:38:49.487133 22989 raft_consensus.cc:340] T 5175d7454e8a47848c04989b12e67f49 P 910125a69a974cec9efdf2fb67e7f9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } I0918 20:38:49.487375 22794 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "b258257413fa48e68e29ef2a9ee9e145" candidate_uuid: "244f0d3980ae451aac40562d8b441de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06662902cf85459a86d2fc0759450bcf" I0918 20:38:49.488255 22794 raft_consensus.cc:2890] T b258257413fa48e68e29ef2a9ee9e145 P 06662902cf85459a86d2fc0759450bcf [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:49.495939 22794 raft_consensus.cc:2350] T b258257413fa48e68e29ef2a9ee9e145 P 06662902cf85459a86d2fc0759450bcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244f0d3980ae451aac40562d8b441de0 in term 1. I0918 20:38:49.497380 22897 leader_election.cc:310] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06662902cf85459a86d2fc0759450bcf, 244f0d3980ae451aac40562d8b441de0; no voters: I0918 20:38:49.498608 22995 raft_consensus.cc:2644] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:49.500341 22995 raft_consensus.cc:667] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [term 1 LEADER]: Becoming Leader. State: Replica: 244f0d3980ae451aac40562d8b441de0, State: Running, Role: LEADER I0918 20:38:49.500964 22995 consensus_queue.cc:206] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } I0918 20:38:49.487673 22989 raft_consensus.cc:366] T 5175d7454e8a47848c04989b12e67f49 P 910125a69a974cec9efdf2fb67e7f9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:49.504806 22989 raft_consensus.cc:705] T 5175d7454e8a47848c04989b12e67f49 P 910125a69a974cec9efdf2fb67e7f9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910125a69a974cec9efdf2fb67e7f9d5, State: Initialized, Role: FOLLOWER I0918 20:38:49.506974 22990 tablet_bootstrap.cc:590] T 5175d7454e8a47848c04989b12e67f49 P 244f0d3980ae451aac40562d8b441de0: No blocks or log segments found. Creating new log. I0918 20:38:49.508652 22865 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "b258257413fa48e68e29ef2a9ee9e145" candidate_uuid: "244f0d3980ae451aac40562d8b441de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "910125a69a974cec9efdf2fb67e7f9d5" W0918 20:38:49.510598 22900 leader_election.cc:348] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 910125a69a974cec9efdf2fb67e7f9d5 (127.18.222.2:40963): Illegal state: must be running to vote when last-logged opid is not known I0918 20:38:49.509147 22994 leader_election.cc:296] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [CANDIDATE]: Term 1 election: Requested vote from peers 06662902cf85459a86d2fc0759450bcf (127.18.222.1:35617), 910125a69a974cec9efdf2fb67e7f9d5 (127.18.222.2:40963) I0918 20:38:49.505578 22989 consensus_queue.cc:229] T 5175d7454e8a47848c04989b12e67f49 P 910125a69a974cec9efdf2fb67e7f9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } I0918 20:38:49.518301 22694 catalog_manager.cc:4131] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 reported cstate change: term changed from 0 to 1, leader changed from to 244f0d3980ae451aac40562d8b441de0 (127.18.222.3). New cstate: current_term: 1 leader_uuid: "244f0d3980ae451aac40562d8b441de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } health_report { overall_health: UNKNOWN } } } I0918 20:38:49.524045 22988 ts_tablet_manager.cc:1113] T 4ecd01ad0981467ba83c8134e92befc0 P 06662902cf85459a86d2fc0759450bcf: Time spent starting tablet: real 0.085s user 0.006s sys 0.000s I0918 20:38:49.525624 22989 ts_tablet_manager.cc:1113] T 5175d7454e8a47848c04989b12e67f49 P 910125a69a974cec9efdf2fb67e7f9d5: Time spent starting tablet: real 0.040s user 0.005s sys 0.000s I0918 20:38:49.534796 22988 tablet_bootstrap.cc:439] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf: Bootstrap starting. I0918 20:38:49.539417 22988 tablet_bootstrap.cc:590] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf: No blocks or log segments found. Creating new log. I0918 20:38:49.544121 22989 tablet_bootstrap.cc:439] T b258257413fa48e68e29ef2a9ee9e145 P 910125a69a974cec9efdf2fb67e7f9d5: Bootstrap starting. I0918 20:38:49.565843 22988 tablet_bootstrap.cc:439] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf: No bootstrap required, opened a new log I0918 20:38:49.567026 22988 ts_tablet_manager.cc:1085] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.004s I0918 20:38:49.567380 22990 tablet_bootstrap.cc:439] T 5175d7454e8a47848c04989b12e67f49 P 244f0d3980ae451aac40562d8b441de0: No bootstrap required, opened a new log I0918 20:38:49.568250 22990 ts_tablet_manager.cc:1085] T 5175d7454e8a47848c04989b12e67f49 P 244f0d3980ae451aac40562d8b441de0: Time spent bootstrapping tablet: real 0.102s user 0.007s sys 0.023s I0918 20:38:49.570639 22990 raft_consensus.cc:340] T 5175d7454e8a47848c04989b12e67f49 P 244f0d3980ae451aac40562d8b441de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } I0918 20:38:49.571208 22990 raft_consensus.cc:366] T 5175d7454e8a47848c04989b12e67f49 P 244f0d3980ae451aac40562d8b441de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:49.571398 22990 raft_consensus.cc:705] T 5175d7454e8a47848c04989b12e67f49 P 244f0d3980ae451aac40562d8b441de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244f0d3980ae451aac40562d8b441de0, State: Initialized, Role: FOLLOWER I0918 20:38:49.572269 22990 consensus_queue.cc:229] T 5175d7454e8a47848c04989b12e67f49 P 244f0d3980ae451aac40562d8b441de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } I0918 20:38:49.563375 22989 tablet_bootstrap.cc:590] T b258257413fa48e68e29ef2a9ee9e145 P 910125a69a974cec9efdf2fb67e7f9d5: No blocks or log segments found. Creating new log. I0918 20:38:49.577911 22990 ts_tablet_manager.cc:1113] T 5175d7454e8a47848c04989b12e67f49 P 244f0d3980ae451aac40562d8b441de0: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s I0918 20:38:49.580551 22988 raft_consensus.cc:340] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } I0918 20:38:49.588380 22988 raft_consensus.cc:366] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:49.603207 22988 raft_consensus.cc:705] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06662902cf85459a86d2fc0759450bcf, State: Initialized, Role: FOLLOWER I0918 20:38:49.604192 22988 consensus_queue.cc:229] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } I0918 20:38:49.604871 22989 tablet_bootstrap.cc:439] T b258257413fa48e68e29ef2a9ee9e145 P 910125a69a974cec9efdf2fb67e7f9d5: No bootstrap required, opened a new log I0918 20:38:49.605346 22989 ts_tablet_manager.cc:1085] T b258257413fa48e68e29ef2a9ee9e145 P 910125a69a974cec9efdf2fb67e7f9d5: Time spent bootstrapping tablet: real 0.064s user 0.013s sys 0.015s I0918 20:38:49.607522 22989 raft_consensus.cc:340] T b258257413fa48e68e29ef2a9ee9e145 P 910125a69a974cec9efdf2fb67e7f9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } I0918 20:38:49.608471 22989 raft_consensus.cc:366] T b258257413fa48e68e29ef2a9ee9e145 P 910125a69a974cec9efdf2fb67e7f9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:49.608781 22989 raft_consensus.cc:705] T b258257413fa48e68e29ef2a9ee9e145 P 910125a69a974cec9efdf2fb67e7f9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910125a69a974cec9efdf2fb67e7f9d5, State: Initialized, Role: FOLLOWER I0918 20:38:49.609553 22989 consensus_queue.cc:229] T b258257413fa48e68e29ef2a9ee9e145 P 910125a69a974cec9efdf2fb67e7f9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } I0918 20:38:49.610545 22988 ts_tablet_manager.cc:1113] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf: Time spent starting tablet: real 0.042s user 0.005s sys 0.000s I0918 20:38:49.617305 22989 ts_tablet_manager.cc:1113] T b258257413fa48e68e29ef2a9ee9e145 P 910125a69a974cec9efdf2fb67e7f9d5: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s I0918 20:38:49.622614 22997 raft_consensus.cc:472] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:49.623102 22997 raft_consensus.cc:494] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } I0918 20:38:49.624769 22997 leader_election.cc:296] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 910125a69a974cec9efdf2fb67e7f9d5 (127.18.222.2:40963), 244f0d3980ae451aac40562d8b441de0 (127.18.222.3:43223) I0918 20:38:49.640507 22996 raft_consensus.cc:472] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:49.641108 22996 raft_consensus.cc:494] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } I0918 20:38:49.643041 22996 leader_election.cc:296] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06662902cf85459a86d2fc0759450bcf (127.18.222.1:35617), 244f0d3980ae451aac40562d8b441de0 (127.18.222.3:43223) I0918 20:38:49.700533 22865 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5175d7454e8a47848c04989b12e67f49" candidate_uuid: "06662902cf85459a86d2fc0759450bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "910125a69a974cec9efdf2fb67e7f9d5" is_pre_election: true I0918 20:38:49.701318 22865 raft_consensus.cc:2350] T 5175d7454e8a47848c04989b12e67f49 P 910125a69a974cec9efdf2fb67e7f9d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06662902cf85459a86d2fc0759450bcf in term 0. I0918 20:38:49.702653 22936 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5175d7454e8a47848c04989b12e67f49" candidate_uuid: "06662902cf85459a86d2fc0759450bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244f0d3980ae451aac40562d8b441de0" is_pre_election: true I0918 20:38:49.703405 22936 raft_consensus.cc:2350] T 5175d7454e8a47848c04989b12e67f49 P 244f0d3980ae451aac40562d8b441de0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06662902cf85459a86d2fc0759450bcf in term 0. I0918 20:38:49.704921 22746 leader_election.cc:310] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06662902cf85459a86d2fc0759450bcf, 910125a69a974cec9efdf2fb67e7f9d5; no voters: I0918 20:38:49.712715 22997 raft_consensus.cc:2644] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:49.713086 22997 raft_consensus.cc:472] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:49.713353 22997 raft_consensus.cc:2890] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:49.721174 22997 raft_consensus.cc:494] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } I0918 20:38:49.723657 22794 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "4ecd01ad0981467ba83c8134e92befc0" candidate_uuid: "910125a69a974cec9efdf2fb67e7f9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06662902cf85459a86d2fc0759450bcf" is_pre_election: true I0918 20:38:49.724452 22794 raft_consensus.cc:2350] T 4ecd01ad0981467ba83c8134e92befc0 P 06662902cf85459a86d2fc0759450bcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910125a69a974cec9efdf2fb67e7f9d5 in term 0. I0918 20:38:49.725473 22865 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5175d7454e8a47848c04989b12e67f49" candidate_uuid: "06662902cf85459a86d2fc0759450bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "910125a69a974cec9efdf2fb67e7f9d5" I0918 20:38:49.725730 22826 leader_election.cc:310] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06662902cf85459a86d2fc0759450bcf, 910125a69a974cec9efdf2fb67e7f9d5; no voters: I0918 20:38:49.725824 22936 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "4ecd01ad0981467ba83c8134e92befc0" candidate_uuid: "910125a69a974cec9efdf2fb67e7f9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244f0d3980ae451aac40562d8b441de0" is_pre_election: true I0918 20:38:49.726091 22865 raft_consensus.cc:2890] T 5175d7454e8a47848c04989b12e67f49 P 910125a69a974cec9efdf2fb67e7f9d5 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:49.726366 22935 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5175d7454e8a47848c04989b12e67f49" candidate_uuid: "06662902cf85459a86d2fc0759450bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244f0d3980ae451aac40562d8b441de0" I0918 20:38:49.726703 22996 raft_consensus.cc:2644] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:49.726477 22936 raft_consensus.cc:2350] T 4ecd01ad0981467ba83c8134e92befc0 P 244f0d3980ae451aac40562d8b441de0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910125a69a974cec9efdf2fb67e7f9d5 in term 0. I0918 20:38:49.727079 22996 raft_consensus.cc:472] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:49.727463 22996 raft_consensus.cc:2890] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:49.727069 22935 raft_consensus.cc:2890] T 5175d7454e8a47848c04989b12e67f49 P 244f0d3980ae451aac40562d8b441de0 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:49.733647 22997 leader_election.cc:296] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf [CANDIDATE]: Term 1 election: Requested vote from peers 910125a69a974cec9efdf2fb67e7f9d5 (127.18.222.2:40963), 244f0d3980ae451aac40562d8b441de0 (127.18.222.3:43223) I0918 20:38:49.733511 22996 raft_consensus.cc:494] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } I0918 20:38:49.733500 22935 raft_consensus.cc:2350] T 5175d7454e8a47848c04989b12e67f49 P 244f0d3980ae451aac40562d8b441de0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06662902cf85459a86d2fc0759450bcf in term 1. I0918 20:38:49.735203 22996 leader_election.cc:296] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 [CANDIDATE]: Term 1 election: Requested vote from peers 06662902cf85459a86d2fc0759450bcf (127.18.222.1:35617), 244f0d3980ae451aac40562d8b441de0 (127.18.222.3:43223) I0918 20:38:49.736194 22935 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "4ecd01ad0981467ba83c8134e92befc0" candidate_uuid: "910125a69a974cec9efdf2fb67e7f9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244f0d3980ae451aac40562d8b441de0" I0918 20:38:49.736655 22935 raft_consensus.cc:2890] T 4ecd01ad0981467ba83c8134e92befc0 P 244f0d3980ae451aac40562d8b441de0 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:49.736899 22794 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "4ecd01ad0981467ba83c8134e92befc0" candidate_uuid: "910125a69a974cec9efdf2fb67e7f9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06662902cf85459a86d2fc0759450bcf" I0918 20:38:49.737512 22794 raft_consensus.cc:2890] T 4ecd01ad0981467ba83c8134e92befc0 P 06662902cf85459a86d2fc0759450bcf [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:49.739965 22746 leader_election.cc:310] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06662902cf85459a86d2fc0759450bcf, 244f0d3980ae451aac40562d8b441de0; no voters: I0918 20:38:49.741055 22997 raft_consensus.cc:2644] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:49.741411 22935 raft_consensus.cc:2350] T 4ecd01ad0981467ba83c8134e92befc0 P 244f0d3980ae451aac40562d8b441de0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910125a69a974cec9efdf2fb67e7f9d5 in term 1. I0918 20:38:49.742537 22829 leader_election.cc:310] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244f0d3980ae451aac40562d8b441de0, 910125a69a974cec9efdf2fb67e7f9d5; no voters: I0918 20:38:49.743501 22996 raft_consensus.cc:2644] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:49.743788 22794 raft_consensus.cc:2350] T 4ecd01ad0981467ba83c8134e92befc0 P 06662902cf85459a86d2fc0759450bcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910125a69a974cec9efdf2fb67e7f9d5 in term 1. I0918 20:38:49.747273 22865 raft_consensus.cc:2350] T 5175d7454e8a47848c04989b12e67f49 P 910125a69a974cec9efdf2fb67e7f9d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06662902cf85459a86d2fc0759450bcf in term 1. I0918 20:38:49.752228 22997 raft_consensus.cc:667] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf [term 1 LEADER]: Becoming Leader. State: Replica: 06662902cf85459a86d2fc0759450bcf, State: Running, Role: LEADER I0918 20:38:49.753057 22996 raft_consensus.cc:667] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 [term 1 LEADER]: Becoming Leader. State: Replica: 910125a69a974cec9efdf2fb67e7f9d5, State: Running, Role: LEADER I0918 20:38:49.753417 22997 consensus_queue.cc:206] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } I0918 20:38:49.753859 22996 consensus_queue.cc:206] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } I0918 20:38:49.762471 22693 catalog_manager.cc:4131] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf reported cstate change: term changed from 0 to 1, leader changed from to 06662902cf85459a86d2fc0759450bcf (127.18.222.1). New cstate: current_term: 1 leader_uuid: "06662902cf85459a86d2fc0759450bcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } health_report { overall_health: UNKNOWN } } } I0918 20:38:49.781739 22692 catalog_manager.cc:4131] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 reported cstate change: term changed from 0 to 1, leader changed from to 910125a69a974cec9efdf2fb67e7f9d5 (127.18.222.2). New cstate: current_term: 1 leader_uuid: "910125a69a974cec9efdf2fb67e7f9d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } health_report { overall_health: UNKNOWN } } } I0918 20:38:49.870620 22996 raft_consensus.cc:472] T b258257413fa48e68e29ef2a9ee9e145 P 910125a69a974cec9efdf2fb67e7f9d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:49.871196 22996 raft_consensus.cc:494] T b258257413fa48e68e29ef2a9ee9e145 P 910125a69a974cec9efdf2fb67e7f9d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 } } peers { permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 } } peers { permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 } } I0918 20:38:49.874045 22794 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "b258257413fa48e68e29ef2a9ee9e145" candidate_uuid: "910125a69a974cec9efdf2fb67e7f9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06662902cf85459a86d2fc0759450bcf" is_pre_election: true I0918 20:38:49.874806 22794 raft_consensus.cc:2280] T b258257413fa48e68e29ef2a9ee9e145 P 06662902cf85459a86d2fc0759450bcf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910125a69a974cec9efdf2fb67e7f9d5 in current term 1: Already voted for candidate 244f0d3980ae451aac40562d8b441de0 in this term. I0918 20:38:49.876263 22996 leader_election.cc:296] T b258257413fa48e68e29ef2a9ee9e145 P 910125a69a974cec9efdf2fb67e7f9d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06662902cf85459a86d2fc0759450bcf (127.18.222.1:35617), 244f0d3980ae451aac40562d8b441de0 (127.18.222.3:43223) I0918 20:38:49.877171 22935 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "b258257413fa48e68e29ef2a9ee9e145" candidate_uuid: "910125a69a974cec9efdf2fb67e7f9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244f0d3980ae451aac40562d8b441de0" is_pre_election: true I0918 20:38:49.877985 22935 raft_consensus.cc:2311] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 910125a69a974cec9efdf2fb67e7f9d5 for term 1 because replica is either leader or believes a valid leader to be alive. I0918 20:38:49.879189 22829 leader_election.cc:310] T b258257413fa48e68e29ef2a9ee9e145 P 910125a69a974cec9efdf2fb67e7f9d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910125a69a974cec9efdf2fb67e7f9d5; no voters: 06662902cf85459a86d2fc0759450bcf, 244f0d3980ae451aac40562d8b441de0 I0918 20:38:49.880913 22996 raft_consensus.cc:2890] T b258257413fa48e68e29ef2a9ee9e145 P 910125a69a974cec9efdf2fb67e7f9d5 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:49.893908 22996 raft_consensus.cc:2592] T b258257413fa48e68e29ef2a9ee9e145 P 910125a69a974cec9efdf2fb67e7f9d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0918 20:38:49.894004 22794 raft_consensus.cc:1184] T b258257413fa48e68e29ef2a9ee9e145 P 06662902cf85459a86d2fc0759450bcf [term 1 FOLLOWER]: Refusing update from remote peer 244f0d3980ae451aac40562d8b441de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:49.894644 22865 raft_consensus.cc:1184] T b258257413fa48e68e29ef2a9ee9e145 P 910125a69a974cec9efdf2fb67e7f9d5 [term 1 FOLLOWER]: Refusing update from remote peer 244f0d3980ae451aac40562d8b441de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:49.895589 22995 consensus_queue.cc:984] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:49.896376 22994 consensus_queue.cc:984] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:50.158300 22935 raft_consensus.cc:1184] T 4ecd01ad0981467ba83c8134e92befc0 P 244f0d3980ae451aac40562d8b441de0 [term 1 FOLLOWER]: Refusing update from remote peer 910125a69a974cec9efdf2fb67e7f9d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:50.158680 22794 raft_consensus.cc:1184] T 4ecd01ad0981467ba83c8134e92befc0 P 06662902cf85459a86d2fc0759450bcf [term 1 FOLLOWER]: Refusing update from remote peer 910125a69a974cec9efdf2fb67e7f9d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:50.160347 23012 consensus_queue.cc:984] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06662902cf85459a86d2fc0759450bcf" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 35617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:50.161603 22996 consensus_queue.cc:984] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:50.170121 22997 consensus_queue.cc:984] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "910125a69a974cec9efdf2fb67e7f9d5" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:50.221401 23011 consensus_queue.cc:984] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "244f0d3980ae451aac40562d8b441de0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0918 20:38:53.258992 23019 outbound_call.cc:287] RPC callback for RPC call kudu.tserver.TabletServerService.Checksum -> {remote=127.18.222.2:40963, user_credentials={real_user=slave}} blocked reactor thread for 86844.4us I0918 20:38:53.283977 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:53.362253 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:53.362640 19320 tablet_replica.cc:273] T 4ecd01ad0981467ba83c8134e92befc0 P 06662902cf85459a86d2fc0759450bcf: stopping tablet replica I0918 20:38:53.363051 19320 raft_consensus.cc:2147] T 4ecd01ad0981467ba83c8134e92befc0 P 06662902cf85459a86d2fc0759450bcf [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:53.363378 19320 raft_consensus.cc:2174] T 4ecd01ad0981467ba83c8134e92befc0 P 06662902cf85459a86d2fc0759450bcf [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:53.365805 19320 tablet_replica.cc:273] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf: stopping tablet replica I0918 20:38:53.366356 19320 raft_consensus.cc:2147] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:53.367287 19320 raft_consensus.cc:2174] T 5175d7454e8a47848c04989b12e67f49 P 06662902cf85459a86d2fc0759450bcf [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:53.371938 19320 tablet_replica.cc:273] T b258257413fa48e68e29ef2a9ee9e145 P 06662902cf85459a86d2fc0759450bcf: stopping tablet replica I0918 20:38:53.372598 19320 raft_consensus.cc:2147] T b258257413fa48e68e29ef2a9ee9e145 P 06662902cf85459a86d2fc0759450bcf [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:53.372999 19320 raft_consensus.cc:2174] T b258257413fa48e68e29ef2a9ee9e145 P 06662902cf85459a86d2fc0759450bcf [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:53.403336 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:53.403599 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:53.404064 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:53.418165 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:53.479533 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:53.479899 19320 tablet_replica.cc:273] T b258257413fa48e68e29ef2a9ee9e145 P 910125a69a974cec9efdf2fb67e7f9d5: stopping tablet replica I0918 20:38:53.480268 19320 raft_consensus.cc:2147] T b258257413fa48e68e29ef2a9ee9e145 P 910125a69a974cec9efdf2fb67e7f9d5 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:53.480530 19320 raft_consensus.cc:2174] T b258257413fa48e68e29ef2a9ee9e145 P 910125a69a974cec9efdf2fb67e7f9d5 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:53.483212 19320 tablet_replica.cc:273] T 5175d7454e8a47848c04989b12e67f49 P 910125a69a974cec9efdf2fb67e7f9d5: stopping tablet replica I0918 20:38:53.483778 19320 raft_consensus.cc:2147] T 5175d7454e8a47848c04989b12e67f49 P 910125a69a974cec9efdf2fb67e7f9d5 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:53.484092 19320 raft_consensus.cc:2174] T 5175d7454e8a47848c04989b12e67f49 P 910125a69a974cec9efdf2fb67e7f9d5 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:53.487071 19320 tablet_replica.cc:273] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5: stopping tablet replica I0918 20:38:53.487458 19320 raft_consensus.cc:2147] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:53.488137 19320 raft_consensus.cc:2174] T 4ecd01ad0981467ba83c8134e92befc0 P 910125a69a974cec9efdf2fb67e7f9d5 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:53.504523 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:53.504770 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:53.505211 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:53.520176 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:53.561369 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:53.561694 19320 tablet_replica.cc:273] T 4ecd01ad0981467ba83c8134e92befc0 P 244f0d3980ae451aac40562d8b441de0: stopping tablet replica I0918 20:38:53.562047 19320 raft_consensus.cc:2147] T 4ecd01ad0981467ba83c8134e92befc0 P 244f0d3980ae451aac40562d8b441de0 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:53.562405 19320 raft_consensus.cc:2174] T 4ecd01ad0981467ba83c8134e92befc0 P 244f0d3980ae451aac40562d8b441de0 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:53.564602 19320 tablet_replica.cc:273] T 5175d7454e8a47848c04989b12e67f49 P 244f0d3980ae451aac40562d8b441de0: stopping tablet replica I0918 20:38:53.564945 19320 raft_consensus.cc:2147] T 5175d7454e8a47848c04989b12e67f49 P 244f0d3980ae451aac40562d8b441de0 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:53.565207 19320 raft_consensus.cc:2174] T 5175d7454e8a47848c04989b12e67f49 P 244f0d3980ae451aac40562d8b441de0 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:53.568918 19320 tablet_replica.cc:273] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0: stopping tablet replica I0918 20:38:53.569257 19320 raft_consensus.cc:2147] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:53.569887 19320 raft_consensus.cc:2174] T b258257413fa48e68e29ef2a9ee9e145 P 244f0d3980ae451aac40562d8b441de0 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:53.591281 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:53.591477 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:53.591859 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:53.604615 19320 master.cc:247] Master@127.18.222.62:38613 shutting down... I0918 20:38:53.627606 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:53.628049 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:53.628316 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 50eed283ebb8415191b2a2ff60a43c25: stopping tablet replica I0918 20:38:53.646594 19320 master.cc:258] Master@127.18.222.62:38613 shutdown complete. I0918 20:38:53.655756 19320 master.cc:247] Master@127.18.222.61:46291 shutting down... I0918 20:38:53.679081 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:53.679502 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:53.679839 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 1fe19deb62c543c3a2f16d29163424de: stopping tablet replica I0918 20:38:53.698094 19320 master.cc:258] Master@127.18.222.61:46291 shutdown complete. W0918 20:38:53.701788 22677 consensus_peers.cc:458] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec -> Peer 1fe19deb62c543c3a2f16d29163424de (127.18.222.61:46291): Couldn't send request to peer 1fe19deb62c543c3a2f16d29163424de. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.61:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:38:53.708413 19320 master.cc:247] Master@127.18.222.60:33817 shutting down... W0918 20:38:53.722671 22678 consensus_peers.cc:458] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec -> Peer 50eed283ebb8415191b2a2ff60a43c25 (127.18.222.62:38613): Couldn't send request to peer 50eed283ebb8415191b2a2ff60a43c25. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.62:38613: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:38:53.733023 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:53.733856 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:53.734107 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P b137a42405ad403182bd6e1b01fc43ec: stopping tablet replica I0918 20:38:53.753438 19320 master.cc:258] Master@127.18.222.60:33817 shutdown complete. [ OK ] RemoteKsckTest.TestChecksumSnapshot (5850 ms) [ RUN ] RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp I0918 20:38:53.789795 19320 test_util.cc:212] Using random seed: -750289335 I0918 20:38:53.790537 19320 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.18.222.62:34771,127.18.222.61:42565,127.18.222.60:40745 I0918 20:38:53.803190 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 144092us I0918 20:38:53.803563 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:53.803751 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:53.803956 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:53.804067 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:53.805101 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/master-0-root/instance: uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" format_stamp: "Formatted at 2019-09-18 20:38:53 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:53.813652 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s I0918 20:38:53.815161 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:53.822028 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:53.822211 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/master-0-root uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" format_stamp: "Formatted at 2019-09-18 20:38:53 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:53.822443 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:53.880040 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:34771 I0918 20:38:53.881778 19320 webserver.cc:374] Webserver started at http://127.18.222.62:38643/ using document root and password file I0918 20:38:53.883463 23129 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:53.888473 23129 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:53.903519 23129 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } has no permanent_uuid. Determining permanent_uuid... W0918 20:38:53.905278 23129 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:42565: Network error: Client connection negotiation failed: client connection to 127.18.222.61:42565: connect: Connection refused (error 111) I0918 20:38:53.907315 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 144092us I0918 20:38:53.907639 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:53.907843 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:53.908031 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:53.908141 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:53.909054 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/master-1-root/instance: uuid: "f41e5b481b104c99945cad9121c4afbf" format_stamp: "Formatted at 2019-09-18 20:38:53 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:53.918632 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.010s sys 0.000s I0918 20:38:53.920164 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:53.927983 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:38:53.928164 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/master-1-root uuid: "f41e5b481b104c99945cad9121c4afbf" format_stamp: "Formatted at 2019-09-18 20:38:53 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:53.928391 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:53.968693 23129 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } attempt: 1 W0918 20:38:53.971112 23129 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:42565: Network error: Client connection negotiation failed: client connection to 127.18.222.61:42565: connect: Connection refused (error 111) I0918 20:38:53.989763 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:42565 I0918 20:38:53.991611 19320 webserver.cc:374] Webserver started at http://127.18.222.61:36693/ using document root and password file I0918 20:38:53.993372 23186 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:54.002682 23186 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:54.013938 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 144592us I0918 20:38:54.014325 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:54.014492 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:54.014716 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:54.014855 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:54.015924 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/master-2-root/instance: uuid: "ef93976569874340bcf974671452ec68" format_stamp: "Formatted at 2019-09-18 20:38:54 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:54.019068 23186 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:54.027525 23129 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } attempt: 2 I0918 20:38:54.031605 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.015s user 0.010s sys 0.005s I0918 20:38:54.033233 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:54.038805 23186 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } has no permanent_uuid. Determining permanent_uuid... W0918 20:38:54.041056 23186 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:40745: Network error: Client connection negotiation failed: client connection to 127.18.222.60:40745: connect: Connection refused (error 111) I0918 20:38:54.043365 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:54.043748 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/master-2-root uuid: "ef93976569874340bcf974671452ec68" format_stamp: "Formatted at 2019-09-18 20:38:54 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:54.044093 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:54.047263 23129 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } has no permanent_uuid. Determining permanent_uuid... W0918 20:38:54.049605 23129 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:40745: Network error: Client connection negotiation failed: client connection to 127.18.222.60:40745: connect: Connection refused (error 111) I0918 20:38:54.087116 23129 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } attempt: 1 W0918 20:38:54.089131 23129 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:40745: Network error: Client connection negotiation failed: client connection to 127.18.222.60:40745: connect: Connection refused (error 111) I0918 20:38:54.100503 23186 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } attempt: 1 W0918 20:38:54.102638 23186 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:40745: Network error: Client connection negotiation failed: client connection to 127.18.222.60:40745: connect: Connection refused (error 111) I0918 20:38:54.156539 23129 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } attempt: 2 W0918 20:38:54.158766 23129 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:40745: Network error: Client connection negotiation failed: client connection to 127.18.222.60:40745: connect: Connection refused (error 111) I0918 20:38:54.164235 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:40745 I0918 20:38:54.165997 19320 webserver.cc:374] Webserver started at http://127.18.222.60:39023/ using document root and password file I0918 20:38:54.166924 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0918 20:38:54.167834 23244 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:54.171067 23186 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } attempt: 2 I0918 20:38:54.173436 23244 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:54.190618 23186 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf: Bootstrap starting. I0918 20:38:54.193346 23244 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:54.193922 23186 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf: No blocks or log segments found. Creating new log. I0918 20:38:54.198084 23186 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf: No bootstrap required, opened a new log I0918 20:38:54.200536 23186 raft_consensus.cc:340] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } I0918 20:38:54.201074 23186 raft_consensus.cc:366] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:54.201272 23186 raft_consensus.cc:705] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41e5b481b104c99945cad9121c4afbf, State: Initialized, Role: FOLLOWER I0918 20:38:54.201995 23186 consensus_queue.cc:229] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } I0918 20:38:54.203936 23186 sys_catalog.cc:340] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } } I0918 20:38:54.204556 23186 sys_catalog.cc:343] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:54.205536 23244 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } has no permanent_uuid. Determining permanent_uuid... I0918 20:38:54.205818 23186 sys_catalog.cc:428] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [sys.catalog]: configured and running, proceeding with master startup. I0918 20:38:54.210835 23248 sys_catalog.cc:340] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } } I0918 20:38:54.212568 23248 sys_catalog.cc:343] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:54.212568 23249 raft_consensus.cc:472] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:54.213459 23249 raft_consensus.cc:494] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } I0918 20:38:54.216046 23249 leader_election.cc:296] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9b53cf41385420d8a156f9f3fdfe2f2 (127.18.222.62:34771), ef93976569874340bcf974671452ec68 (127.18.222.60:40745) W0918 20:38:54.217358 23250 catalog_manager.cc:1155] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:38:54.217695 23250 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:54.227330 23244 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68: Bootstrap starting. I0918 20:38:54.231163 23244 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68: No blocks or log segments found. Creating new log. I0918 20:38:54.233219 23129 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } attempt: 3 I0918 20:38:54.235600 23244 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68: No bootstrap required, opened a new log I0918 20:38:54.238212 23244 raft_consensus.cc:340] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } I0918 20:38:54.239001 23244 raft_consensus.cc:366] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:54.239249 23244 raft_consensus.cc:705] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef93976569874340bcf974671452ec68, State: Initialized, Role: FOLLOWER I0918 20:38:54.239945 23244 consensus_queue.cc:229] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } I0918 20:38:54.241928 23244 sys_catalog.cc:340] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } } I0918 20:38:54.242679 23244 sys_catalog.cc:343] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:54.242223 23252 sys_catalog.cc:340] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } } I0918 20:38:54.243100 23252 sys_catalog.cc:343] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:54.244046 23244 sys_catalog.cc:428] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:38:54.244199 23221 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f41e5b481b104c99945cad9121c4afbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef93976569874340bcf974671452ec68" is_pre_election: true I0918 20:38:54.245424 23221 raft_consensus.cc:2350] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f41e5b481b104c99945cad9121c4afbf in term 0. I0918 20:38:54.246714 23135 leader_election.cc:310] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ef93976569874340bcf974671452ec68, f41e5b481b104c99945cad9121c4afbf; no voters: I0918 20:38:54.247792 23249 raft_consensus.cc:2644] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:54.248095 23249 raft_consensus.cc:472] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:54.248395 23249 raft_consensus.cc:2890] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:54.254689 23249 raft_consensus.cc:494] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } I0918 20:38:54.256657 23129 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2: Bootstrap starting. W0918 20:38:54.259147 23253 catalog_manager.cc:1155] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68: acquiring CA information for follower catalog manager: Not found: root CA entry not found I0918 20:38:54.259465 23129 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2: No blocks or log segments found. Creating new log. I0918 20:38:54.260550 23249 leader_election.cc:296] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [CANDIDATE]: Term 1 election: Requested vote from peers b9b53cf41385420d8a156f9f3fdfe2f2 (127.18.222.62:34771), ef93976569874340bcf974671452ec68 (127.18.222.60:40745) W0918 20:38:54.259461 23253 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:54.261584 23221 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f41e5b481b104c99945cad9121c4afbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef93976569874340bcf974671452ec68" I0918 20:38:54.262243 23221 raft_consensus.cc:2890] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:54.269691 23129 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2: No bootstrap required, opened a new log I0918 20:38:54.272306 23221 raft_consensus.cc:2350] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f41e5b481b104c99945cad9121c4afbf in term 1. I0918 20:38:54.271607 23129 raft_consensus.cc:340] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } I0918 20:38:54.273407 23135 leader_election.cc:310] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ef93976569874340bcf974671452ec68, f41e5b481b104c99945cad9121c4afbf; no voters: I0918 20:38:54.274183 23249 raft_consensus.cc:2644] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:54.274194 23129 raft_consensus.cc:366] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:54.274690 23129 raft_consensus.cc:705] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9b53cf41385420d8a156f9f3fdfe2f2, State: Initialized, Role: FOLLOWER I0918 20:38:54.274789 23249 raft_consensus.cc:667] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [term 1 LEADER]: Becoming Leader. State: Replica: f41e5b481b104c99945cad9121c4afbf, State: Running, Role: LEADER I0918 20:38:54.275327 23129 consensus_queue.cc:229] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } I0918 20:38:54.275470 23249 consensus_queue.cc:206] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } I0918 20:38:54.276856 23129 sys_catalog.cc:340] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } } I0918 20:38:54.277112 23255 sys_catalog.cc:340] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } } I0918 20:38:54.277653 23129 sys_catalog.cc:343] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:54.277907 23255 sys_catalog.cc:343] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:54.279155 23129 sys_catalog.cc:428] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:38:54.279435 23105 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f41e5b481b104c99945cad9121c4afbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" I0918 20:38:54.279350 23106 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f41e5b481b104c99945cad9121c4afbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" is_pre_election: true I0918 20:38:54.280215 23105 raft_consensus.cc:2890] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:54.280148 23248 sys_catalog.cc:340] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f41e5b481b104c99945cad9121c4afbf. Latest consensus state: current_term: 1 leader_uuid: "f41e5b481b104c99945cad9121c4afbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } } I0918 20:38:54.281270 23248 sys_catalog.cc:343] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [sys.catalog]: This master's current role is: LEADER I0918 20:38:54.284111 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 I0918 20:38:54.284445 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 I0918 20:38:54.284886 23258 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:38:54.287195 23105 raft_consensus.cc:2350] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f41e5b481b104c99945cad9121c4afbf in term 1. I0918 20:38:54.287407 23258 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:38:54.287848 23106 raft_consensus.cc:2325] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f41e5b481b104c99945cad9121c4afbf for term 1 because replica is already servicing an update from a current leader or another vote. W0918 20:38:54.293105 23257 catalog_manager.cc:1155] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:38:54.293455 23257 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:38:54.302475 23106 raft_consensus.cc:1184] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2 [term 1 FOLLOWER]: Refusing update from remote peer f41e5b481b104c99945cad9121c4afbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:54.302502 23221 raft_consensus.cc:1184] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68 [term 1 FOLLOWER]: Refusing update from remote peer f41e5b481b104c99945cad9121c4afbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:54.304227 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 144592us I0918 20:38:54.304085 23248 consensus_queue.cc:984] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:54.304733 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:54.304980 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:54.304811 23249 consensus_queue.cc:984] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:54.305245 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:54.305464 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:54.306865 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/ts-0-root/instance: uuid: "d7935ea8243841c6a5744c979d50d58c" format_stamp: "Formatted at 2019-09-18 20:38:54 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:54.316776 23252 sys_catalog.cc:340] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f41e5b481b104c99945cad9121c4afbf. Latest consensus state: current_term: 1 leader_uuid: "f41e5b481b104c99945cad9121c4afbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } } I0918 20:38:54.317728 23252 sys_catalog.cc:343] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:54.319228 23255 sys_catalog.cc:340] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f41e5b481b104c99945cad9121c4afbf. Latest consensus state: current_term: 1 leader_uuid: "f41e5b481b104c99945cad9121c4afbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } } I0918 20:38:54.319957 23255 sys_catalog.cc:343] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:54.322505 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.015s user 0.010s sys 0.003s I0918 20:38:54.322374 23249 sys_catalog.cc:340] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f41e5b481b104c99945cad9121c4afbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } } I0918 20:38:54.323257 23249 sys_catalog.cc:343] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [sys.catalog]: This master's current role is: LEADER I0918 20:38:54.324734 23258 catalog_manager.cc:981] Generated new certificate authority record I0918 20:38:54.326236 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:54.327457 23258 catalog_manager.cc:1106] Loading token signing keys... I0918 20:38:54.328889 23252 sys_catalog.cc:340] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f41e5b481b104c99945cad9121c4afbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } } I0918 20:38:54.329799 23252 sys_catalog.cc:343] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:54.333914 23249 sys_catalog.cc:340] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f41e5b481b104c99945cad9121c4afbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } } I0918 20:38:54.342356 23249 sys_catalog.cc:343] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [sys.catalog]: This master's current role is: LEADER I0918 20:38:54.343660 23255 sys_catalog.cc:340] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f41e5b481b104c99945cad9121c4afbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b53cf41385420d8a156f9f3fdfe2f2" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34771 } } peers { permanent_uuid: "f41e5b481b104c99945cad9121c4afbf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42565 } } peers { permanent_uuid: "ef93976569874340bcf974671452ec68" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 40745 } } } I0918 20:38:54.344540 23255 sys_catalog.cc:343] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:38:54.349460 19320 fs_manager.cc:419] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s I0918 20:38:54.349768 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/ts-0-root uuid: "d7935ea8243841c6a5744c979d50d58c" format_stamp: "Formatted at 2019-09-18 20:38:54 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:54.350039 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:54.359980 23258 catalog_manager.cc:4446] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf: Generated new TSK 0 I0918 20:38:54.360857 23258 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:38:54.413693 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:54.413892 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:54.453055 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:38227 I0918 20:38:54.454826 19320 webserver.cc:374] Webserver started at http://127.18.222.1:34563/ using document root and password file I0918 20:38:54.482240 23333 heartbeater.cc:322] Connected to a master server at 127.18.222.60:40745 I0918 20:38:54.482604 23333 heartbeater.cc:406] Registering TS with master... I0918 20:38:54.483184 23333 heartbeater.cc:455] Master 127.18.222.60:40745 requested a full tablet report, sending... I0918 20:38:54.483283 23334 heartbeater.cc:322] Connected to a master server at 127.18.222.61:42565 I0918 20:38:54.483749 23334 heartbeater.cc:406] Registering TS with master... I0918 20:38:54.484329 23334 heartbeater.cc:455] Master 127.18.222.61:42565 requested a full tablet report, sending... I0918 20:38:54.486068 23211 ts_manager.cc:186] Registered new tserver with Master: d7935ea8243841c6a5744c979d50d58c (127.18.222.1:38227) I0918 20:38:54.486407 23153 ts_manager.cc:186] Registered new tserver with Master: d7935ea8243841c6a5744c979d50d58c (127.18.222.1:38227) I0918 20:38:54.488692 23153 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33010 I0918 20:38:54.493079 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 144592us I0918 20:38:54.493413 23335 heartbeater.cc:322] Connected to a master server at 127.18.222.62:34771 I0918 20:38:54.493510 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:54.493780 23335 heartbeater.cc:406] Registering TS with master... I0918 20:38:54.493827 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:54.494154 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:54.494287 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:54.494189 23335 heartbeater.cc:455] Master 127.18.222.62:34771 requested a full tablet report, sending... I0918 20:38:54.495550 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/ts-1-root/instance: uuid: "c411f7a565ba4ce98b932b3fa45d63a3" format_stamp: "Formatted at 2019-09-18 20:38:54 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:54.496567 23096 ts_manager.cc:186] Registered new tserver with Master: d7935ea8243841c6a5744c979d50d58c (127.18.222.1:38227) I0918 20:38:54.501919 23334 heartbeater.cc:447] Master 127.18.222.61:42565 was elected leader, sending a full tablet report... I0918 20:38:54.506088 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.010s sys 0.000s I0918 20:38:54.507756 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:54.517390 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:54.517587 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/ts-1-root uuid: "c411f7a565ba4ce98b932b3fa45d63a3" format_stamp: "Formatted at 2019-09-18 20:38:54 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:54.517844 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:54.573292 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:54.573493 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:54.624080 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.2:46507 I0918 20:38:54.627801 19320 webserver.cc:374] Webserver started at http://127.18.222.2:38673/ using document root and password file I0918 20:38:54.648123 23404 heartbeater.cc:322] Connected to a master server at 127.18.222.60:40745 I0918 20:38:54.648538 23404 heartbeater.cc:406] Registering TS with master... I0918 20:38:54.649130 23404 heartbeater.cc:455] Master 127.18.222.60:40745 requested a full tablet report, sending... I0918 20:38:54.652118 23211 ts_manager.cc:186] Registered new tserver with Master: c411f7a565ba4ce98b932b3fa45d63a3 (127.18.222.2:46507) I0918 20:38:54.654160 23405 heartbeater.cc:322] Connected to a master server at 127.18.222.61:42565 I0918 20:38:54.654593 23405 heartbeater.cc:406] Registering TS with master... I0918 20:38:54.655194 23405 heartbeater.cc:455] Master 127.18.222.61:42565 requested a full tablet report, sending... I0918 20:38:54.656875 23153 ts_manager.cc:186] Registered new tserver with Master: c411f7a565ba4ce98b932b3fa45d63a3 (127.18.222.2:46507) I0918 20:38:54.659217 23153 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33016 I0918 20:38:54.660045 23406 heartbeater.cc:322] Connected to a master server at 127.18.222.62:34771 I0918 20:38:54.660392 23406 heartbeater.cc:406] Registering TS with master... I0918 20:38:54.660892 23406 heartbeater.cc:455] Master 127.18.222.62:34771 requested a full tablet report, sending... I0918 20:38:54.663630 23096 ts_manager.cc:186] Registered new tserver with Master: c411f7a565ba4ce98b932b3fa45d63a3 (127.18.222.2:46507) I0918 20:38:54.669246 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 144592us I0918 20:38:54.669662 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:54.669831 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:54.670049 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:54.670184 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:54.671211 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/ts-2-root/instance: uuid: "cbe7176ee05f4ea98a8c284ebb25709f" format_stamp: "Formatted at 2019-09-18 20:38:54 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:54.674175 23405 heartbeater.cc:447] Master 127.18.222.61:42565 was elected leader, sending a full tablet report... I0918 20:38:54.684656 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.013s user 0.004s sys 0.012s I0918 20:38:54.686419 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:54.699687 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:54.699980 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/ts-2-root uuid: "cbe7176ee05f4ea98a8c284ebb25709f" format_stamp: "Formatted at 2019-09-18 20:38:54 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:54.700263 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1568839089958813-19320/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:38:54.734268 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:38:54.734467 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:38:54.788736 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.3:43541 I0918 20:38:54.791100 19320 webserver.cc:374] Webserver started at http://127.18.222.3:34873/ using document root and password file I0918 20:38:54.824062 23475 heartbeater.cc:322] Connected to a master server at 127.18.222.60:40745 I0918 20:38:54.824434 23475 heartbeater.cc:406] Registering TS with master... I0918 20:38:54.825012 23475 heartbeater.cc:455] Master 127.18.222.60:40745 requested a full tablet report, sending... I0918 20:38:54.827549 23210 ts_manager.cc:186] Registered new tserver with Master: cbe7176ee05f4ea98a8c284ebb25709f (127.18.222.3:43541) I0918 20:38:54.839684 23476 heartbeater.cc:322] Connected to a master server at 127.18.222.61:42565 I0918 20:38:54.840095 23476 heartbeater.cc:406] Registering TS with master... I0918 20:38:54.840644 23476 heartbeater.cc:455] Master 127.18.222.61:42565 requested a full tablet report, sending... I0918 20:38:54.842845 23478 heartbeater.cc:322] Connected to a master server at 127.18.222.62:34771 I0918 20:38:54.843189 23478 heartbeater.cc:406] Registering TS with master... I0918 20:38:54.843262 23152 ts_manager.cc:186] Registered new tserver with Master: cbe7176ee05f4ea98a8c284ebb25709f (127.18.222.3:43541) I0918 20:38:54.843902 23478 heartbeater.cc:455] Master 127.18.222.62:34771 requested a full tablet report, sending... I0918 20:38:54.846141 23152 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33022 I0918 20:38:54.849464 23095 ts_manager.cc:186] Registered new tserver with Master: cbe7176ee05f4ea98a8c284ebb25709f (127.18.222.3:43541) I0918 20:38:54.849680 19320 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.031194827s I0918 20:38:54.859949 23476 heartbeater.cc:447] Master 127.18.222.61:42565 was elected leader, sending a full tablet report... I0918 20:38:54.912863 23153 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33030: name: "ksck-test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\n\000\000\000\004\001\024\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:38:54.914697 23153 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:38:55.089916 23300 tablet_service.cc:1022] Processing CreateTablet for tablet 31651063e45a4279a645826f3fbc7ec2 (table=ksck-test-table [id=e48ff36693f34829bce3035496b4c397]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:55.091123 23300 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 31651063e45a4279a645826f3fbc7ec2. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:55.098588 23299 tablet_service.cc:1022] Processing CreateTablet for tablet 6301319767b84b0abf0c7a142df9ad56 (table=ksck-test-table [id=e48ff36693f34829bce3035496b4c397]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:55.099825 23299 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 6301319767b84b0abf0c7a142df9ad56. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:55.104817 23298 tablet_service.cc:1022] Processing CreateTablet for tablet 3a083531e57347b483543fabfd76ed48 (table=ksck-test-table [id=e48ff36693f34829bce3035496b4c397]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:55.105919 23298 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 3a083531e57347b483543fabfd76ed48. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:55.114589 23442 tablet_service.cc:1022] Processing CreateTablet for tablet 31651063e45a4279a645826f3fbc7ec2 (table=ksck-test-table [id=e48ff36693f34829bce3035496b4c397]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:55.115679 23442 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 31651063e45a4279a645826f3fbc7ec2. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:55.122289 23371 tablet_service.cc:1022] Processing CreateTablet for tablet 31651063e45a4279a645826f3fbc7ec2 (table=ksck-test-table [id=e48ff36693f34829bce3035496b4c397]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:38:55.123476 23371 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 31651063e45a4279a645826f3fbc7ec2. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:55.123646 23441 tablet_service.cc:1022] Processing CreateTablet for tablet 6301319767b84b0abf0c7a142df9ad56 (table=ksck-test-table [id=e48ff36693f34829bce3035496b4c397]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:55.124441 23370 tablet_service.cc:1022] Processing CreateTablet for tablet 6301319767b84b0abf0c7a142df9ad56 (table=ksck-test-table [id=e48ff36693f34829bce3035496b4c397]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:38:55.124805 23441 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 6301319767b84b0abf0c7a142df9ad56. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:55.125613 23369 tablet_service.cc:1022] Processing CreateTablet for tablet 3a083531e57347b483543fabfd76ed48 (table=ksck-test-table [id=e48ff36693f34829bce3035496b4c397]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:55.126582 23369 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 3a083531e57347b483543fabfd76ed48. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:55.131213 23370 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 6301319767b84b0abf0c7a142df9ad56. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:55.137359 23440 tablet_service.cc:1022] Processing CreateTablet for tablet 3a083531e57347b483543fabfd76ed48 (table=ksck-test-table [id=e48ff36693f34829bce3035496b4c397]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:38:55.138496 23440 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 3a083531e57347b483543fabfd76ed48. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:38:55.171813 23504 tablet_bootstrap.cc:439] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c: Bootstrap starting. I0918 20:38:55.175683 23504 tablet_bootstrap.cc:590] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c: No blocks or log segments found. Creating new log. I0918 20:38:55.187463 23505 tablet_bootstrap.cc:439] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3: Bootstrap starting. I0918 20:38:55.193367 23505 tablet_bootstrap.cc:590] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3: No blocks or log segments found. Creating new log. I0918 20:38:55.214268 23506 tablet_bootstrap.cc:439] T 6301319767b84b0abf0c7a142df9ad56 P cbe7176ee05f4ea98a8c284ebb25709f: Bootstrap starting. I0918 20:38:55.221750 23504 tablet_bootstrap.cc:439] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c: No bootstrap required, opened a new log I0918 20:38:55.222338 23504 ts_tablet_manager.cc:1085] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c: Time spent bootstrapping tablet: real 0.051s user 0.011s sys 0.027s I0918 20:38:55.225741 23504 raft_consensus.cc:340] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.226522 23504 raft_consensus.cc:366] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:55.226758 23504 raft_consensus.cc:705] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7935ea8243841c6a5744c979d50d58c, State: Initialized, Role: FOLLOWER I0918 20:38:55.229025 23504 consensus_queue.cc:229] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.219636 23506 tablet_bootstrap.cc:590] T 6301319767b84b0abf0c7a142df9ad56 P cbe7176ee05f4ea98a8c284ebb25709f: No blocks or log segments found. Creating new log. I0918 20:38:55.237242 23505 tablet_bootstrap.cc:439] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3: No bootstrap required, opened a new log I0918 20:38:55.237761 23505 ts_tablet_manager.cc:1085] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.027s I0918 20:38:55.239326 23506 tablet_bootstrap.cc:439] T 6301319767b84b0abf0c7a142df9ad56 P cbe7176ee05f4ea98a8c284ebb25709f: No bootstrap required, opened a new log I0918 20:38:55.242336 23506 ts_tablet_manager.cc:1085] T 6301319767b84b0abf0c7a142df9ad56 P cbe7176ee05f4ea98a8c284ebb25709f: Time spent bootstrapping tablet: real 0.056s user 0.009s sys 0.000s I0918 20:38:55.243023 23505 raft_consensus.cc:340] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.243825 23505 raft_consensus.cc:366] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:55.244038 23505 raft_consensus.cc:705] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c411f7a565ba4ce98b932b3fa45d63a3, State: Initialized, Role: FOLLOWER I0918 20:38:55.244313 23506 raft_consensus.cc:340] T 6301319767b84b0abf0c7a142df9ad56 P cbe7176ee05f4ea98a8c284ebb25709f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.245163 23506 raft_consensus.cc:366] T 6301319767b84b0abf0c7a142df9ad56 P cbe7176ee05f4ea98a8c284ebb25709f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:55.245399 23506 raft_consensus.cc:705] T 6301319767b84b0abf0c7a142df9ad56 P cbe7176ee05f4ea98a8c284ebb25709f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbe7176ee05f4ea98a8c284ebb25709f, State: Initialized, Role: FOLLOWER I0918 20:38:55.246183 23506 consensus_queue.cc:229] T 6301319767b84b0abf0c7a142df9ad56 P cbe7176ee05f4ea98a8c284ebb25709f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.255357 23504 ts_tablet_manager.cc:1113] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c: Time spent starting tablet: real 0.033s user 0.009s sys 0.002s I0918 20:38:55.256016 23505 consensus_queue.cc:229] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.257925 23504 tablet_bootstrap.cc:439] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c: Bootstrap starting. I0918 20:38:55.267724 23505 ts_tablet_manager.cc:1113] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3: Time spent starting tablet: real 0.030s user 0.001s sys 0.007s I0918 20:38:55.268599 23506 ts_tablet_manager.cc:1113] T 6301319767b84b0abf0c7a142df9ad56 P cbe7176ee05f4ea98a8c284ebb25709f: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s I0918 20:38:55.268627 23505 tablet_bootstrap.cc:439] T 3a083531e57347b483543fabfd76ed48 P c411f7a565ba4ce98b932b3fa45d63a3: Bootstrap starting. I0918 20:38:55.269605 23506 tablet_bootstrap.cc:439] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f: Bootstrap starting. I0918 20:38:55.273231 23505 tablet_bootstrap.cc:590] T 3a083531e57347b483543fabfd76ed48 P c411f7a565ba4ce98b932b3fa45d63a3: No blocks or log segments found. Creating new log. I0918 20:38:55.273504 23506 tablet_bootstrap.cc:590] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f: No blocks or log segments found. Creating new log. I0918 20:38:55.278967 23504 tablet_bootstrap.cc:590] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c: No blocks or log segments found. Creating new log. I0918 20:38:55.281167 23253 catalog_manager.cc:1153] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68: acquiring CA information for follower catalog manager: success I0918 20:38:55.291942 23511 raft_consensus.cc:472] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:55.292551 23511 raft_consensus.cc:494] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.294740 23511 leader_election.cc:296] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7935ea8243841c6a5744c979d50d58c (127.18.222.1:38227), cbe7176ee05f4ea98a8c284ebb25709f (127.18.222.3:43541) I0918 20:38:55.373786 23257 catalog_manager.cc:1153] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2: acquiring CA information for follower catalog manager: success I0918 20:38:55.385612 23505 tablet_bootstrap.cc:439] T 3a083531e57347b483543fabfd76ed48 P c411f7a565ba4ce98b932b3fa45d63a3: No bootstrap required, opened a new log I0918 20:38:55.386122 23505 ts_tablet_manager.cc:1085] T 3a083531e57347b483543fabfd76ed48 P c411f7a565ba4ce98b932b3fa45d63a3: Time spent bootstrapping tablet: real 0.118s user 0.017s sys 0.032s I0918 20:38:55.388803 23310 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6301319767b84b0abf0c7a142df9ad56" candidate_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7935ea8243841c6a5744c979d50d58c" is_pre_election: true I0918 20:38:55.388744 23506 tablet_bootstrap.cc:439] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f: No bootstrap required, opened a new log I0918 20:38:55.389423 23253 catalog_manager.cc:1181] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:38:55.389688 23506 ts_tablet_manager.cc:1085] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f: Time spent bootstrapping tablet: real 0.120s user 0.022s sys 0.057s W0918 20:38:55.390441 23345 leader_election.cc:348] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7935ea8243841c6a5744c979d50d58c (127.18.222.1:38227): Illegal state: must be running to vote when last-logged opid is not known I0918 20:38:55.392062 23506 raft_consensus.cc:340] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.392833 23506 raft_consensus.cc:366] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:55.393064 23506 raft_consensus.cc:705] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbe7176ee05f4ea98a8c284ebb25709f, State: Initialized, Role: FOLLOWER I0918 20:38:55.393630 23506 consensus_queue.cc:229] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.395596 23506 ts_tablet_manager.cc:1113] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s I0918 20:38:55.396387 23506 tablet_bootstrap.cc:439] T 3a083531e57347b483543fabfd76ed48 P cbe7176ee05f4ea98a8c284ebb25709f: Bootstrap starting. I0918 20:38:55.400823 23504 tablet_bootstrap.cc:439] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c: No bootstrap required, opened a new log I0918 20:38:55.401906 23504 ts_tablet_manager.cc:1085] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c: Time spent bootstrapping tablet: real 0.144s user 0.037s sys 0.037s I0918 20:38:55.411984 23505 raft_consensus.cc:340] T 3a083531e57347b483543fabfd76ed48 P c411f7a565ba4ce98b932b3fa45d63a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.412413 23257 catalog_manager.cc:1181] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:38:55.415014 23505 raft_consensus.cc:366] T 3a083531e57347b483543fabfd76ed48 P c411f7a565ba4ce98b932b3fa45d63a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:55.416728 23505 raft_consensus.cc:705] T 3a083531e57347b483543fabfd76ed48 P c411f7a565ba4ce98b932b3fa45d63a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c411f7a565ba4ce98b932b3fa45d63a3, State: Initialized, Role: FOLLOWER I0918 20:38:55.417594 23505 consensus_queue.cc:229] T 3a083531e57347b483543fabfd76ed48 P c411f7a565ba4ce98b932b3fa45d63a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.421785 23506 tablet_bootstrap.cc:590] T 3a083531e57347b483543fabfd76ed48 P cbe7176ee05f4ea98a8c284ebb25709f: No blocks or log segments found. Creating new log. I0918 20:38:55.409687 23504 raft_consensus.cc:340] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.424221 23504 raft_consensus.cc:366] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:55.424489 23504 raft_consensus.cc:705] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7935ea8243841c6a5744c979d50d58c, State: Initialized, Role: FOLLOWER I0918 20:38:55.424608 23505 ts_tablet_manager.cc:1113] T 3a083531e57347b483543fabfd76ed48 P c411f7a565ba4ce98b932b3fa45d63a3: Time spent starting tablet: real 0.038s user 0.004s sys 0.004s I0918 20:38:55.425540 23505 tablet_bootstrap.cc:439] T 31651063e45a4279a645826f3fbc7ec2 P c411f7a565ba4ce98b932b3fa45d63a3: Bootstrap starting. I0918 20:38:55.425187 23504 consensus_queue.cc:229] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.427528 23452 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6301319767b84b0abf0c7a142df9ad56" candidate_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" is_pre_election: true I0918 20:38:55.428194 23452 raft_consensus.cc:2350] T 6301319767b84b0abf0c7a142df9ad56 P cbe7176ee05f4ea98a8c284ebb25709f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c411f7a565ba4ce98b932b3fa45d63a3 in term 0. I0918 20:38:55.429661 23504 ts_tablet_manager.cc:1113] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c: Time spent starting tablet: real 0.027s user 0.004s sys 0.002s I0918 20:38:55.430496 23505 tablet_bootstrap.cc:590] T 31651063e45a4279a645826f3fbc7ec2 P c411f7a565ba4ce98b932b3fa45d63a3: No blocks or log segments found. Creating new log. I0918 20:38:55.439657 23343 leader_election.cc:310] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c411f7a565ba4ce98b932b3fa45d63a3, cbe7176ee05f4ea98a8c284ebb25709f; no voters: d7935ea8243841c6a5744c979d50d58c I0918 20:38:55.440433 23511 raft_consensus.cc:2644] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:55.440737 23511 raft_consensus.cc:472] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:55.441048 23511 raft_consensus.cc:2890] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:55.447100 23512 raft_consensus.cc:472] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:55.448436 23512 raft_consensus.cc:494] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.430507 23504 tablet_bootstrap.cc:439] T 6301319767b84b0abf0c7a142df9ad56 P d7935ea8243841c6a5744c979d50d58c: Bootstrap starting. I0918 20:38:55.463626 23504 tablet_bootstrap.cc:590] T 6301319767b84b0abf0c7a142df9ad56 P d7935ea8243841c6a5744c979d50d58c: No blocks or log segments found. Creating new log. I0918 20:38:55.450505 23512 leader_election.cc:296] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7935ea8243841c6a5744c979d50d58c (127.18.222.1:38227), c411f7a565ba4ce98b932b3fa45d63a3 (127.18.222.2:46507) I0918 20:38:55.499454 23505 tablet_bootstrap.cc:439] T 31651063e45a4279a645826f3fbc7ec2 P c411f7a565ba4ce98b932b3fa45d63a3: No bootstrap required, opened a new log I0918 20:38:55.500416 23505 ts_tablet_manager.cc:1085] T 31651063e45a4279a645826f3fbc7ec2 P c411f7a565ba4ce98b932b3fa45d63a3: Time spent bootstrapping tablet: real 0.075s user 0.018s sys 0.004s I0918 20:38:55.504755 23510 raft_consensus.cc:472] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:55.505368 23510 raft_consensus.cc:494] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.509450 23505 raft_consensus.cc:340] T 31651063e45a4279a645826f3fbc7ec2 P c411f7a565ba4ce98b932b3fa45d63a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.529516 23505 raft_consensus.cc:366] T 31651063e45a4279a645826f3fbc7ec2 P c411f7a565ba4ce98b932b3fa45d63a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:55.541718 23506 tablet_bootstrap.cc:439] T 3a083531e57347b483543fabfd76ed48 P cbe7176ee05f4ea98a8c284ebb25709f: No bootstrap required, opened a new log I0918 20:38:55.542526 23510 leader_election.cc:296] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c411f7a565ba4ce98b932b3fa45d63a3 (127.18.222.2:46507), cbe7176ee05f4ea98a8c284ebb25709f (127.18.222.3:43541) I0918 20:38:55.543835 23505 raft_consensus.cc:705] T 31651063e45a4279a645826f3fbc7ec2 P c411f7a565ba4ce98b932b3fa45d63a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c411f7a565ba4ce98b932b3fa45d63a3, State: Initialized, Role: FOLLOWER I0918 20:38:55.542400 23511 raft_consensus.cc:494] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.558090 23504 tablet_bootstrap.cc:439] T 6301319767b84b0abf0c7a142df9ad56 P d7935ea8243841c6a5744c979d50d58c: No bootstrap required, opened a new log I0918 20:38:55.558810 23504 ts_tablet_manager.cc:1085] T 6301319767b84b0abf0c7a142df9ad56 P d7935ea8243841c6a5744c979d50d58c: Time spent bootstrapping tablet: real 0.128s user 0.022s sys 0.016s I0918 20:38:55.556931 23505 consensus_queue.cc:229] T 31651063e45a4279a645826f3fbc7ec2 P c411f7a565ba4ce98b932b3fa45d63a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.560762 23452 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6301319767b84b0abf0c7a142df9ad56" candidate_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" I0918 20:38:55.561380 23452 raft_consensus.cc:2890] T 6301319767b84b0abf0c7a142df9ad56 P cbe7176ee05f4ea98a8c284ebb25709f [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:55.561908 23511 leader_election.cc:296] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [CANDIDATE]: Term 1 election: Requested vote from peers d7935ea8243841c6a5744c979d50d58c (127.18.222.1:38227), cbe7176ee05f4ea98a8c284ebb25709f (127.18.222.3:43541) I0918 20:38:55.567586 23452 raft_consensus.cc:2350] T 6301319767b84b0abf0c7a142df9ad56 P cbe7176ee05f4ea98a8c284ebb25709f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c411f7a565ba4ce98b932b3fa45d63a3 in term 1. I0918 20:38:55.572388 23343 leader_election.cc:310] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c411f7a565ba4ce98b932b3fa45d63a3, cbe7176ee05f4ea98a8c284ebb25709f; no voters: I0918 20:38:55.576424 23511 raft_consensus.cc:2644] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:55.576869 23511 raft_consensus.cc:667] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [term 1 LEADER]: Becoming Leader. State: Replica: c411f7a565ba4ce98b932b3fa45d63a3, State: Running, Role: LEADER I0918 20:38:55.578629 23506 ts_tablet_manager.cc:1085] T 3a083531e57347b483543fabfd76ed48 P cbe7176ee05f4ea98a8c284ebb25709f: Time spent bootstrapping tablet: real 0.182s user 0.011s sys 0.016s I0918 20:38:55.592948 23310 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6301319767b84b0abf0c7a142df9ad56" candidate_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7935ea8243841c6a5744c979d50d58c" I0918 20:38:55.594156 23504 raft_consensus.cc:340] T 6301319767b84b0abf0c7a142df9ad56 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.595752 23506 raft_consensus.cc:340] T 3a083531e57347b483543fabfd76ed48 P cbe7176ee05f4ea98a8c284ebb25709f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } W0918 20:38:55.596719 23345 leader_election.cc:348] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d7935ea8243841c6a5744c979d50d58c (127.18.222.1:38227): Illegal state: must be running to vote when last-logged opid is not known I0918 20:38:55.598742 23506 raft_consensus.cc:366] T 3a083531e57347b483543fabfd76ed48 P cbe7176ee05f4ea98a8c284ebb25709f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:55.584923 23511 consensus_queue.cc:206] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.599025 23506 raft_consensus.cc:705] T 3a083531e57347b483543fabfd76ed48 P cbe7176ee05f4ea98a8c284ebb25709f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbe7176ee05f4ea98a8c284ebb25709f, State: Initialized, Role: FOLLOWER I0918 20:38:55.599874 23506 consensus_queue.cc:229] T 3a083531e57347b483543fabfd76ed48 P cbe7176ee05f4ea98a8c284ebb25709f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.603171 23506 ts_tablet_manager.cc:1113] T 3a083531e57347b483543fabfd76ed48 P cbe7176ee05f4ea98a8c284ebb25709f: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s I0918 20:38:55.598265 23505 ts_tablet_manager.cc:1113] T 31651063e45a4279a645826f3fbc7ec2 P c411f7a565ba4ce98b932b3fa45d63a3: Time spent starting tablet: real 0.098s user 0.010s sys 0.011s I0918 20:38:55.598641 23504 raft_consensus.cc:366] T 6301319767b84b0abf0c7a142df9ad56 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:38:55.613546 23504 raft_consensus.cc:705] T 6301319767b84b0abf0c7a142df9ad56 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7935ea8243841c6a5744c979d50d58c, State: Initialized, Role: FOLLOWER I0918 20:38:55.614300 23504 consensus_queue.cc:229] T 6301319767b84b0abf0c7a142df9ad56 P d7935ea8243841c6a5744c979d50d58c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.618053 23504 ts_tablet_manager.cc:1113] T 6301319767b84b0abf0c7a142df9ad56 P d7935ea8243841c6a5744c979d50d58c: Time spent starting tablet: real 0.059s user 0.006s sys 0.000s I0918 20:38:55.632550 23381 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "31651063e45a4279a645826f3fbc7ec2" candidate_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" is_pre_election: true I0918 20:38:55.633456 23381 raft_consensus.cc:2350] T 31651063e45a4279a645826f3fbc7ec2 P c411f7a565ba4ce98b932b3fa45d63a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbe7176ee05f4ea98a8c284ebb25709f in term 0. I0918 20:38:55.635929 23415 leader_election.cc:310] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c411f7a565ba4ce98b932b3fa45d63a3, cbe7176ee05f4ea98a8c284ebb25709f; no voters: I0918 20:38:55.636991 23512 raft_consensus.cc:2644] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:55.637375 23512 raft_consensus.cc:472] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:55.637701 23512 raft_consensus.cc:2890] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:55.638088 23381 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "31651063e45a4279a645826f3fbc7ec2" candidate_uuid: "d7935ea8243841c6a5744c979d50d58c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" is_pre_election: true I0918 20:38:55.638801 23381 raft_consensus.cc:2350] T 31651063e45a4279a645826f3fbc7ec2 P c411f7a565ba4ce98b932b3fa45d63a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7935ea8243841c6a5744c979d50d58c in term 0. I0918 20:38:55.642920 23273 leader_election.cc:310] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c411f7a565ba4ce98b932b3fa45d63a3, d7935ea8243841c6a5744c979d50d58c; no voters: I0918 20:38:55.643647 23510 raft_consensus.cc:2644] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:55.644062 23510 raft_consensus.cc:472] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:55.644289 23510 raft_consensus.cc:2890] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:55.650641 23510 raft_consensus.cc:494] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.652042 23310 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "31651063e45a4279a645826f3fbc7ec2" candidate_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7935ea8243841c6a5744c979d50d58c" is_pre_election: true I0918 20:38:55.653091 23310 raft_consensus.cc:2280] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cbe7176ee05f4ea98a8c284ebb25709f in current term 1: Already voted for candidate d7935ea8243841c6a5744c979d50d58c in this term. I0918 20:38:55.653648 23381 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "31651063e45a4279a645826f3fbc7ec2" candidate_uuid: "d7935ea8243841c6a5744c979d50d58c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" I0918 20:38:55.654222 23381 raft_consensus.cc:2890] T 31651063e45a4279a645826f3fbc7ec2 P c411f7a565ba4ce98b932b3fa45d63a3 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:55.656386 23512 raft_consensus.cc:494] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.658905 23310 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "31651063e45a4279a645826f3fbc7ec2" candidate_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7935ea8243841c6a5744c979d50d58c" I0918 20:38:55.659622 23310 raft_consensus.cc:2280] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cbe7176ee05f4ea98a8c284ebb25709f in current term 1: Already voted for candidate d7935ea8243841c6a5744c979d50d58c in this term. I0918 20:38:55.652449 23510 leader_election.cc:296] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [CANDIDATE]: Term 1 election: Requested vote from peers c411f7a565ba4ce98b932b3fa45d63a3 (127.18.222.2:46507), cbe7176ee05f4ea98a8c284ebb25709f (127.18.222.3:43541) I0918 20:38:55.665007 23381 raft_consensus.cc:2350] T 31651063e45a4279a645826f3fbc7ec2 P c411f7a565ba4ce98b932b3fa45d63a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7935ea8243841c6a5744c979d50d58c in term 1. I0918 20:38:55.667053 23273 leader_election.cc:310] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c411f7a565ba4ce98b932b3fa45d63a3, d7935ea8243841c6a5744c979d50d58c; no voters: I0918 20:38:55.668028 23510 raft_consensus.cc:2644] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:55.668656 23512 leader_election.cc:296] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f [CANDIDATE]: Term 1 election: Requested vote from peers d7935ea8243841c6a5744c979d50d58c (127.18.222.1:38227), c411f7a565ba4ce98b932b3fa45d63a3 (127.18.222.2:46507) I0918 20:38:55.669279 23381 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "31651063e45a4279a645826f3fbc7ec2" candidate_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" I0918 20:38:55.668951 23153 catalog_manager.cc:4131] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 reported cstate change: term changed from 0 to 1, leader changed from to c411f7a565ba4ce98b932b3fa45d63a3 (127.18.222.2). New cstate: current_term: 1 leader_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } health_report { overall_health: UNKNOWN } } } I0918 20:38:55.670230 23381 raft_consensus.cc:2280] T 31651063e45a4279a645826f3fbc7ec2 P c411f7a565ba4ce98b932b3fa45d63a3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cbe7176ee05f4ea98a8c284ebb25709f in current term 1: Already voted for candidate d7935ea8243841c6a5744c979d50d58c in this term. I0918 20:38:55.671277 23415 leader_election.cc:310] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cbe7176ee05f4ea98a8c284ebb25709f; no voters: c411f7a565ba4ce98b932b3fa45d63a3, d7935ea8243841c6a5744c979d50d58c I0918 20:38:55.676045 23452 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "31651063e45a4279a645826f3fbc7ec2" candidate_uuid: "d7935ea8243841c6a5744c979d50d58c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" is_pre_election: true I0918 20:38:55.676445 23512 raft_consensus.cc:2592] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority I0918 20:38:55.677026 23452 raft_consensus.cc:2280] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7935ea8243841c6a5744c979d50d58c in current term 1: Already voted for candidate cbe7176ee05f4ea98a8c284ebb25709f in this term. I0918 20:38:55.674888 23510 raft_consensus.cc:667] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [term 1 LEADER]: Becoming Leader. State: Replica: d7935ea8243841c6a5744c979d50d58c, State: Running, Role: LEADER I0918 20:38:55.680022 23452 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "31651063e45a4279a645826f3fbc7ec2" candidate_uuid: "d7935ea8243841c6a5744c979d50d58c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" I0918 20:38:55.680982 23452 raft_consensus.cc:2280] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d7935ea8243841c6a5744c979d50d58c in current term 1: Already voted for candidate cbe7176ee05f4ea98a8c284ebb25709f in this term. I0918 20:38:55.681802 23510 consensus_queue.cc:206] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } W0918 20:38:55.770298 23136 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.222.62:34771, user_credentials={real_user=slave}} blocked reactor thread for 43706us I0918 20:38:55.774823 23151 catalog_manager.cc:4131] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c reported cstate change: term changed from 0 to 1, leader changed from to d7935ea8243841c6a5744c979d50d58c (127.18.222.1). New cstate: current_term: 1 leader_uuid: "d7935ea8243841c6a5744c979d50d58c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } health_report { overall_health: UNKNOWN } } } I0918 20:38:55.823242 23510 raft_consensus.cc:472] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:55.823913 23510 raft_consensus.cc:494] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.825747 23510 leader_election.cc:296] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c411f7a565ba4ce98b932b3fa45d63a3 (127.18.222.2:46507), cbe7176ee05f4ea98a8c284ebb25709f (127.18.222.3:43541) I0918 20:38:55.826359 23381 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3a083531e57347b483543fabfd76ed48" candidate_uuid: "d7935ea8243841c6a5744c979d50d58c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" is_pre_election: true I0918 20:38:55.827095 23381 raft_consensus.cc:2350] T 3a083531e57347b483543fabfd76ed48 P c411f7a565ba4ce98b932b3fa45d63a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7935ea8243841c6a5744c979d50d58c in term 0. I0918 20:38:55.827175 23452 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3a083531e57347b483543fabfd76ed48" candidate_uuid: "d7935ea8243841c6a5744c979d50d58c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" is_pre_election: true I0918 20:38:55.827852 23452 raft_consensus.cc:2350] T 3a083531e57347b483543fabfd76ed48 P cbe7176ee05f4ea98a8c284ebb25709f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7935ea8243841c6a5744c979d50d58c in term 0. I0918 20:38:55.828294 23273 leader_election.cc:310] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c411f7a565ba4ce98b932b3fa45d63a3, d7935ea8243841c6a5744c979d50d58c; no voters: I0918 20:38:55.829192 23510 raft_consensus.cc:2644] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:38:55.829488 23510 raft_consensus.cc:472] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:38:55.829787 23510 raft_consensus.cc:2890] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:55.836478 23510 raft_consensus.cc:494] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.838580 23381 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3a083531e57347b483543fabfd76ed48" candidate_uuid: "d7935ea8243841c6a5744c979d50d58c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" I0918 20:38:55.839167 23381 raft_consensus.cc:2890] T 3a083531e57347b483543fabfd76ed48 P c411f7a565ba4ce98b932b3fa45d63a3 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:55.845124 23381 raft_consensus.cc:2350] T 3a083531e57347b483543fabfd76ed48 P c411f7a565ba4ce98b932b3fa45d63a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7935ea8243841c6a5744c979d50d58c in term 1. I0918 20:38:55.846189 23273 leader_election.cc:310] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c411f7a565ba4ce98b932b3fa45d63a3, d7935ea8243841c6a5744c979d50d58c; no voters: I0918 20:38:55.846526 23510 leader_election.cc:296] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c [CANDIDATE]: Term 1 election: Requested vote from peers c411f7a565ba4ce98b932b3fa45d63a3 (127.18.222.2:46507), cbe7176ee05f4ea98a8c284ebb25709f (127.18.222.3:43541) I0918 20:38:55.847270 23452 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3a083531e57347b483543fabfd76ed48" candidate_uuid: "d7935ea8243841c6a5744c979d50d58c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" I0918 20:38:55.850919 23452 raft_consensus.cc:2890] T 3a083531e57347b483543fabfd76ed48 P cbe7176ee05f4ea98a8c284ebb25709f [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:55.851184 23528 raft_consensus.cc:2644] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:38:55.858453 23452 raft_consensus.cc:2350] T 3a083531e57347b483543fabfd76ed48 P cbe7176ee05f4ea98a8c284ebb25709f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7935ea8243841c6a5744c979d50d58c in term 1. I0918 20:38:55.861728 23528 raft_consensus.cc:667] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c [term 1 LEADER]: Becoming Leader. State: Replica: d7935ea8243841c6a5744c979d50d58c, State: Running, Role: LEADER I0918 20:38:55.862511 23528 consensus_queue.cc:206] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.872599 23151 catalog_manager.cc:4131] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c reported cstate change: term changed from 0 to 1, leader changed from to d7935ea8243841c6a5744c979d50d58c (127.18.222.1). New cstate: current_term: 1 leader_uuid: "d7935ea8243841c6a5744c979d50d58c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } health_report { overall_health: UNKNOWN } } } I0918 20:38:55.988991 23528 raft_consensus.cc:472] T 6301319767b84b0abf0c7a142df9ad56 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:38:55.989603 23528 raft_consensus.cc:494] T 6301319767b84b0abf0c7a142df9ad56 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 } } peers { permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 } } peers { permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 } } I0918 20:38:55.991477 23528 leader_election.cc:296] T 6301319767b84b0abf0c7a142df9ad56 P d7935ea8243841c6a5744c979d50d58c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c411f7a565ba4ce98b932b3fa45d63a3 (127.18.222.2:46507), cbe7176ee05f4ea98a8c284ebb25709f (127.18.222.3:43541) I0918 20:38:55.992368 23381 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6301319767b84b0abf0c7a142df9ad56" candidate_uuid: "d7935ea8243841c6a5744c979d50d58c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" is_pre_election: true I0918 20:38:55.992553 23452 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6301319767b84b0abf0c7a142df9ad56" candidate_uuid: "d7935ea8243841c6a5744c979d50d58c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" is_pre_election: true I0918 20:38:55.993189 23381 raft_consensus.cc:2311] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate d7935ea8243841c6a5744c979d50d58c for term 1 because replica is either leader or believes a valid leader to be alive. I0918 20:38:55.993351 23452 raft_consensus.cc:2280] T 6301319767b84b0abf0c7a142df9ad56 P cbe7176ee05f4ea98a8c284ebb25709f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7935ea8243841c6a5744c979d50d58c in current term 1: Already voted for candidate c411f7a565ba4ce98b932b3fa45d63a3 in this term. I0918 20:38:55.995115 23273 leader_election.cc:310] T 6301319767b84b0abf0c7a142df9ad56 P d7935ea8243841c6a5744c979d50d58c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7935ea8243841c6a5744c979d50d58c; no voters: c411f7a565ba4ce98b932b3fa45d63a3, cbe7176ee05f4ea98a8c284ebb25709f I0918 20:38:55.995802 23528 raft_consensus.cc:2890] T 6301319767b84b0abf0c7a142df9ad56 P d7935ea8243841c6a5744c979d50d58c [term 0 FOLLOWER]: Advancing to term 1 I0918 20:38:55.999570 23452 raft_consensus.cc:1184] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f [term 1 FOLLOWER]: Refusing update from remote peer d7935ea8243841c6a5744c979d50d58c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:56.003296 23381 raft_consensus.cc:1184] T 31651063e45a4279a645826f3fbc7ec2 P c411f7a565ba4ce98b932b3fa45d63a3 [term 1 FOLLOWER]: Refusing update from remote peer d7935ea8243841c6a5744c979d50d58c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:38:56.004381 23510 consensus_queue.cc:984] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:56.005208 23539 consensus_queue.cc:984] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:56.020979 23528 raft_consensus.cc:2592] T 6301319767b84b0abf0c7a142df9ad56 P d7935ea8243841c6a5744c979d50d58c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0918 20:38:56.048125 23549 mvcc.cc:203] Tried to move safe_time back from 6425965101029269504 to 6425965099758026752. Current Snapshot: MvccSnapshot[committed={T|T < 6425965101029269504}] I0918 20:38:56.189465 23511 consensus_queue.cc:984] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7935ea8243841c6a5744c979d50d58c" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:56.219820 23511 consensus_queue.cc:984] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:38:56.403656 23528 consensus_queue.cc:984] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c411f7a565ba4ce98b932b3fa45d63a3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:38:56.441373 23528 consensus_queue.cc:984] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbe7176ee05f4ea98a8c284ebb25709f" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 43541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0918 20:38:57.514873 23272 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.222.3:43541, user_credentials={real_user=slave}} blocked reactor thread for 54178.6us I0918 20:38:59.502863 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:59.570674 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:59.571039 19320 tablet_replica.cc:273] T 6301319767b84b0abf0c7a142df9ad56 P d7935ea8243841c6a5744c979d50d58c: stopping tablet replica I0918 20:38:59.571430 19320 raft_consensus.cc:2147] T 6301319767b84b0abf0c7a142df9ad56 P d7935ea8243841c6a5744c979d50d58c [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:59.571776 19320 raft_consensus.cc:2174] T 6301319767b84b0abf0c7a142df9ad56 P d7935ea8243841c6a5744c979d50d58c [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:59.573935 19320 tablet_replica.cc:273] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c: stopping tablet replica I0918 20:38:59.574429 19320 raft_consensus.cc:2147] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:59.575587 19320 raft_consensus.cc:2174] T 3a083531e57347b483543fabfd76ed48 P d7935ea8243841c6a5744c979d50d58c [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:59.578732 19320 tablet_replica.cc:273] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c: stopping tablet replica I0918 20:38:59.579071 19320 raft_consensus.cc:2147] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:59.579871 19320 raft_consensus.cc:2174] T 31651063e45a4279a645826f3fbc7ec2 P d7935ea8243841c6a5744c979d50d58c [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:59.606099 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:59.606312 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:59.606631 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:59.620350 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:59.664921 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:59.665421 19320 tablet_replica.cc:273] T 31651063e45a4279a645826f3fbc7ec2 P c411f7a565ba4ce98b932b3fa45d63a3: stopping tablet replica I0918 20:38:59.665976 19320 raft_consensus.cc:2147] T 31651063e45a4279a645826f3fbc7ec2 P c411f7a565ba4ce98b932b3fa45d63a3 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:59.666350 19320 raft_consensus.cc:2174] T 31651063e45a4279a645826f3fbc7ec2 P c411f7a565ba4ce98b932b3fa45d63a3 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:59.668931 19320 tablet_replica.cc:273] T 3a083531e57347b483543fabfd76ed48 P c411f7a565ba4ce98b932b3fa45d63a3: stopping tablet replica I0918 20:38:59.669418 19320 raft_consensus.cc:2147] T 3a083531e57347b483543fabfd76ed48 P c411f7a565ba4ce98b932b3fa45d63a3 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:59.669785 19320 raft_consensus.cc:2174] T 3a083531e57347b483543fabfd76ed48 P c411f7a565ba4ce98b932b3fa45d63a3 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:59.672526 19320 tablet_replica.cc:273] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3: stopping tablet replica I0918 20:38:59.672886 19320 raft_consensus.cc:2147] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:59.673501 19320 raft_consensus.cc:2174] T 6301319767b84b0abf0c7a142df9ad56 P c411f7a565ba4ce98b932b3fa45d63a3 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:59.697882 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:59.698067 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:59.698443 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:59.712170 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:59.744313 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:38:59.744587 19320 tablet_replica.cc:273] T 3a083531e57347b483543fabfd76ed48 P cbe7176ee05f4ea98a8c284ebb25709f: stopping tablet replica I0918 20:38:59.745029 19320 raft_consensus.cc:2147] T 3a083531e57347b483543fabfd76ed48 P cbe7176ee05f4ea98a8c284ebb25709f [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:59.745357 19320 raft_consensus.cc:2174] T 3a083531e57347b483543fabfd76ed48 P cbe7176ee05f4ea98a8c284ebb25709f [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:59.748379 19320 tablet_replica.cc:273] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f: stopping tablet replica I0918 20:38:59.748715 19320 raft_consensus.cc:2147] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:59.748957 19320 raft_consensus.cc:2174] T 31651063e45a4279a645826f3fbc7ec2 P cbe7176ee05f4ea98a8c284ebb25709f [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:59.750705 19320 tablet_replica.cc:273] T 6301319767b84b0abf0c7a142df9ad56 P cbe7176ee05f4ea98a8c284ebb25709f: stopping tablet replica I0918 20:38:59.750999 19320 raft_consensus.cc:2147] T 6301319767b84b0abf0c7a142df9ad56 P cbe7176ee05f4ea98a8c284ebb25709f [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:59.751232 19320 raft_consensus.cc:2174] T 6301319767b84b0abf0c7a142df9ad56 P cbe7176ee05f4ea98a8c284ebb25709f [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:59.771875 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:59.772066 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:38:59.772392 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:38:59.783576 19320 master.cc:247] Master@127.18.222.62:34771 shutting down... I0918 20:38:59.805996 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:59.806375 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:59.806764 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P b9b53cf41385420d8a156f9f3fdfe2f2: stopping tablet replica I0918 20:38:59.824781 19320 master.cc:258] Master@127.18.222.62:34771 shutdown complete. I0918 20:38:59.832818 19320 master.cc:247] Master@127.18.222.61:42565 shutting down... I0918 20:38:59.853113 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [term 1 LEADER]: Raft consensus shutting down. I0918 20:38:59.853876 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:59.854123 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P f41e5b481b104c99945cad9121c4afbf: stopping tablet replica I0918 20:38:59.872396 19320 master.cc:258] Master@127.18.222.61:42565 shutdown complete. I0918 20:38:59.883985 19320 master.cc:247] Master@127.18.222.60:40745 shutting down... I0918 20:38:59.903025 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:38:59.903549 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:38:59.903846 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P ef93976569874340bcf974671452ec68: stopping tablet replica I0918 20:38:59.921322 19320 master.cc:258] Master@127.18.222.60:40745 shutdown complete. [ OK ] RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp (6162 ms) [ RUN ] RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM I0918 20:38:59.952343 19320 test_util.cc:212] Using random seed: -744126789 I0918 20:38:59.953095 19320 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.18.222.62:34677,127.18.222.61:33575,127.18.222.60:32807 I0918 20:38:59.965700 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 147092us I0918 20:38:59.966099 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:38:59.966274 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:38:59.966492 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:38:59.966619 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:38:59.967644 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/master-0-root/instance: uuid: "9677e12125ca44d483fd9cab2182a2a0" format_stamp: "Formatted at 2019-09-18 20:38:59 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:59.976554 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.005s sys 0.003s I0918 20:38:59.978090 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:38:59.984931 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:38:59.985160 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/master-0-root uuid: "9677e12125ca44d483fd9cab2182a2a0" format_stamp: "Formatted at 2019-09-18 20:38:59 on dist-test-slave-dist-test-slave-85lk" I0918 20:38:59.985424 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:00.034711 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:34677 I0918 20:39:00.036509 19320 webserver.cc:374] Webserver started at http://127.18.222.62:32919/ using document root and password file I0918 20:39:00.038192 23659 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:00.043099 23659 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:00.062708 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 147592us I0918 20:39:00.063302 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:00.063594 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:00.063961 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:00.064079 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:00.065317 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/master-1-root/instance: uuid: "5149d1f41d644da7ad63a91236c71d5b" format_stamp: "Formatted at 2019-09-18 20:39:00 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:00.067538 23659 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:00.069353 23659 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:33575: Network error: Client connection negotiation failed: client connection to 127.18.222.61:33575: connect: Connection refused (error 111) I0918 20:39:00.075945 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.008s sys 0.004s I0918 20:39:00.077598 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:00.085852 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:39:00.086053 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/master-1-root uuid: "5149d1f41d644da7ad63a91236c71d5b" format_stamp: "Formatted at 2019-09-18 20:39:00 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:00.086354 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:00.117719 23659 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } attempt: 1 W0918 20:39:00.119787 23659 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:33575: Network error: Client connection negotiation failed: client connection to 127.18.222.61:33575: connect: Connection refused (error 111) I0918 20:39:00.154147 23659 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } attempt: 2 W0918 20:39:00.156230 23659 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:33575: Network error: Client connection negotiation failed: client connection to 127.18.222.61:33575: connect: Connection refused (error 111) I0918 20:39:00.163600 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:33575 I0918 20:39:00.165498 19320 webserver.cc:374] Webserver started at http://127.18.222.61:45321/ using document root and password file I0918 20:39:00.167165 23716 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:00.172207 23716 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:00.187525 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 147592us I0918 20:39:00.188001 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:00.188257 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:00.188491 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:00.188625 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:00.189476 23716 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:00.189761 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/master-2-root/instance: uuid: "120945e9b78f41feaa64417f83cdee59" format_stamp: "Formatted at 2019-09-18 20:39:00 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:00.200634 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.006s sys 0.004s I0918 20:39:00.201897 23716 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:00.202826 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 W0918 20:39:00.203888 23716 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:32807: Network error: Client connection negotiation failed: client connection to 127.18.222.60:32807: connect: Connection refused (error 111) I0918 20:39:00.211084 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:00.211361 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/master-2-root uuid: "120945e9b78f41feaa64417f83cdee59" format_stamp: "Formatted at 2019-09-18 20:39:00 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:00.211598 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:00.226613 23659 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } attempt: 3 I0918 20:39:00.239477 23659 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:00.241302 23659 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:32807: Network error: Client connection negotiation failed: client connection to 127.18.222.60:32807: connect: Connection refused (error 111) I0918 20:39:00.257448 23716 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } attempt: 1 W0918 20:39:00.260300 23716 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:32807: Network error: Client connection negotiation failed: client connection to 127.18.222.60:32807: connect: Connection refused (error 111) I0918 20:39:00.264806 23659 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } attempt: 1 W0918 20:39:00.267446 23659 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:32807: Network error: Client connection negotiation failed: client connection to 127.18.222.60:32807: connect: Connection refused (error 111) I0918 20:39:00.270325 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:32807 I0918 20:39:00.272238 19320 webserver.cc:374] Webserver started at http://127.18.222.60:38037/ using document root and password file I0918 20:39:00.273216 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0918 20:39:00.274032 23773 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:00.278695 23773 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:00.294519 23773 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:00.304576 23773 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:00.319842 23773 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59: Bootstrap starting. I0918 20:39:00.321976 23659 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } attempt: 2 I0918 20:39:00.322494 23773 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59: No blocks or log segments found. Creating new log. I0918 20:39:00.324782 23716 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } attempt: 2 I0918 20:39:00.326797 23773 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59: No bootstrap required, opened a new log I0918 20:39:00.328732 23773 raft_consensus.cc:340] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } I0918 20:39:00.329457 23773 raft_consensus.cc:366] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:00.329690 23773 raft_consensus.cc:705] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 120945e9b78f41feaa64417f83cdee59, State: Initialized, Role: FOLLOWER I0918 20:39:00.330345 23773 consensus_queue.cc:229] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } I0918 20:39:00.332458 23773 sys_catalog.cc:340] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } } I0918 20:39:00.333210 23773 sys_catalog.cc:343] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:00.332698 23778 sys_catalog.cc:340] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } } I0918 20:39:00.333763 23778 sys_catalog.cc:343] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:00.334318 23773 sys_catalog.cc:428] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59 [sys.catalog]: configured and running, proceeding with master startup. W0918 20:39:00.337741 23779 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:00.337957 23779 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:00.343257 23659 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0: Bootstrap starting. I0918 20:39:00.347085 23659 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0: No blocks or log segments found. Creating new log. I0918 20:39:00.347723 23716 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b: Bootstrap starting. I0918 20:39:00.350782 23659 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0: No bootstrap required, opened a new log I0918 20:39:00.351430 23716 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b: No blocks or log segments found. Creating new log. I0918 20:39:00.353175 23659 raft_consensus.cc:340] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } I0918 20:39:00.354002 23659 raft_consensus.cc:366] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:00.354218 23659 raft_consensus.cc:705] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9677e12125ca44d483fd9cab2182a2a0, State: Initialized, Role: FOLLOWER I0918 20:39:00.354899 23716 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b: No bootstrap required, opened a new log I0918 20:39:00.354898 23659 consensus_queue.cc:229] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } I0918 20:39:00.357003 23782 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } } I0918 20:39:00.356904 23659 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } } I0918 20:39:00.357625 23782 sys_catalog.cc:343] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:00.357695 23659 sys_catalog.cc:343] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:00.357384 23716 raft_consensus.cc:340] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } I0918 20:39:00.358144 23716 raft_consensus.cc:366] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:00.358413 23716 raft_consensus.cc:705] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5149d1f41d644da7ad63a91236c71d5b, State: Initialized, Role: FOLLOWER I0918 20:39:00.358970 23659 sys_catalog.cc:428] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:00.358996 23716 consensus_queue.cc:229] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } I0918 20:39:00.360342 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 W0918 20:39:00.362329 23783 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:00.362573 23783 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:00.363158 23716 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } } I0918 20:39:00.363301 23784 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } } I0918 20:39:00.363734 23716 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:00.363930 23784 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:00.364694 23716 sys_catalog.cc:428] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:00.365691 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 W0918 20:39:00.368336 23785 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:00.368649 23785 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:00.381815 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 147592us I0918 20:39:00.382218 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:00.382380 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:00.382598 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:00.382722 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:00.383805 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/ts-0-root/instance: uuid: "0b23c65819784c9d8689114664470d10" format_stamp: "Formatted at 2019-09-18 20:39:00 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:00.386528 23784 raft_consensus.cc:472] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:00.387042 23784 raft_consensus.cc:494] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } I0918 20:39:00.388501 23784 leader_election.cc:296] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9677e12125ca44d483fd9cab2182a2a0 (127.18.222.62:34677), 120945e9b78f41feaa64417f83cdee59 (127.18.222.60:32807) I0918 20:39:00.389575 23750 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5149d1f41d644da7ad63a91236c71d5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "120945e9b78f41feaa64417f83cdee59" is_pre_election: true I0918 20:39:00.389583 23636 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5149d1f41d644da7ad63a91236c71d5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9677e12125ca44d483fd9cab2182a2a0" is_pre_election: true I0918 20:39:00.390314 23750 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5149d1f41d644da7ad63a91236c71d5b in term 0. I0918 20:39:00.390396 23636 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5149d1f41d644da7ad63a91236c71d5b in term 0. I0918 20:39:00.391335 23664 leader_election.cc:310] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 120945e9b78f41feaa64417f83cdee59, 5149d1f41d644da7ad63a91236c71d5b; no voters: I0918 20:39:00.392074 23784 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:00.392388 23784 raft_consensus.cc:472] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:00.392642 23784 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:00.395967 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.012s user 0.011s sys 0.001s I0918 20:39:00.397183 23784 raft_consensus.cc:494] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } I0918 20:39:00.397598 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:00.398555 23784 leader_election.cc:296] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [CANDIDATE]: Term 1 election: Requested vote from peers 9677e12125ca44d483fd9cab2182a2a0 (127.18.222.62:34677), 120945e9b78f41feaa64417f83cdee59 (127.18.222.60:32807) I0918 20:39:00.399593 23750 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5149d1f41d644da7ad63a91236c71d5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "120945e9b78f41feaa64417f83cdee59" I0918 20:39:00.399588 23636 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5149d1f41d644da7ad63a91236c71d5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9677e12125ca44d483fd9cab2182a2a0" I0918 20:39:00.400442 23750 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:00.400492 23636 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:00.406603 23750 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5149d1f41d644da7ad63a91236c71d5b in term 1. I0918 20:39:00.406599 23636 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5149d1f41d644da7ad63a91236c71d5b in term 1. I0918 20:39:00.408080 19320 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s I0918 20:39:00.408115 23664 leader_election.cc:310] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 120945e9b78f41feaa64417f83cdee59, 5149d1f41d644da7ad63a91236c71d5b; no voters: I0918 20:39:00.408466 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/ts-0-root uuid: "0b23c65819784c9d8689114664470d10" format_stamp: "Formatted at 2019-09-18 20:39:00 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:00.408777 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:00.409024 23784 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:00.410360 23784 raft_consensus.cc:667] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [term 1 LEADER]: Becoming Leader. State: Replica: 5149d1f41d644da7ad63a91236c71d5b, State: Running, Role: LEADER I0918 20:39:00.410992 23784 consensus_queue.cc:206] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } I0918 20:39:00.415360 23788 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5149d1f41d644da7ad63a91236c71d5b. Latest consensus state: current_term: 1 leader_uuid: "5149d1f41d644da7ad63a91236c71d5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } } I0918 20:39:00.416164 23788 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [sys.catalog]: This master's current role is: LEADER I0918 20:39:00.420756 23794 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:39:00.423362 23794 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:39:00.442567 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:00.442757 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:00.497296 23636 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0 [term 1 FOLLOWER]: Refusing update from remote peer 5149d1f41d644da7ad63a91236c71d5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:00.497597 23750 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59 [term 1 FOLLOWER]: Refusing update from remote peer 5149d1f41d644da7ad63a91236c71d5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:00.499464 23788 consensus_queue.cc:984] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:39:00.500878 23784 consensus_queue.cc:984] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:00.514757 23782 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5149d1f41d644da7ad63a91236c71d5b. Latest consensus state: current_term: 1 leader_uuid: "5149d1f41d644da7ad63a91236c71d5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } } I0918 20:39:00.516440 23782 sys_catalog.cc:343] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:00.519803 23778 sys_catalog.cc:340] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5149d1f41d644da7ad63a91236c71d5b. Latest consensus state: current_term: 1 leader_uuid: "5149d1f41d644da7ad63a91236c71d5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } } I0918 20:39:00.520486 23778 sys_catalog.cc:343] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:00.527209 23788 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5149d1f41d644da7ad63a91236c71d5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } } I0918 20:39:00.528254 23788 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [sys.catalog]: This master's current role is: LEADER I0918 20:39:00.531761 23782 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5149d1f41d644da7ad63a91236c71d5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } } I0918 20:39:00.531882 23788 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5149d1f41d644da7ad63a91236c71d5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } } I0918 20:39:00.532768 23782 sys_catalog.cc:343] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:00.532996 23788 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [sys.catalog]: This master's current role is: LEADER I0918 20:39:00.533416 23794 catalog_manager.cc:981] Generated new certificate authority record I0918 20:39:00.535457 23794 catalog_manager.cc:1106] Loading token signing keys... I0918 20:39:00.552768 23778 sys_catalog.cc:340] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5149d1f41d644da7ad63a91236c71d5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9677e12125ca44d483fd9cab2182a2a0" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34677 } } peers { permanent_uuid: "5149d1f41d644da7ad63a91236c71d5b" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 33575 } } peers { permanent_uuid: "120945e9b78f41feaa64417f83cdee59" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32807 } } } I0918 20:39:00.553932 23778 sys_catalog.cc:343] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:00.565065 23794 catalog_manager.cc:4446] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b: Generated new TSK 0 I0918 20:39:00.565980 23794 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:39:00.587496 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:43635 I0918 20:39:00.589962 19320 webserver.cc:374] Webserver started at http://127.18.222.1:42437/ using document root and password file I0918 20:39:00.624997 23863 heartbeater.cc:322] Connected to a master server at 127.18.222.60:32807 I0918 20:39:00.625505 23863 heartbeater.cc:406] Registering TS with master... I0918 20:39:00.626243 23863 heartbeater.cc:455] Master 127.18.222.60:32807 requested a full tablet report, sending... I0918 20:39:00.629644 23740 ts_manager.cc:186] Registered new tserver with Master: 0b23c65819784c9d8689114664470d10 (127.18.222.1:43635) I0918 20:39:00.630059 23864 heartbeater.cc:322] Connected to a master server at 127.18.222.61:33575 I0918 20:39:00.630493 23864 heartbeater.cc:406] Registering TS with master... I0918 20:39:00.631126 23864 heartbeater.cc:455] Master 127.18.222.61:33575 requested a full tablet report, sending... I0918 20:39:00.631749 23865 heartbeater.cc:322] Connected to a master server at 127.18.222.62:34677 I0918 20:39:00.632110 23865 heartbeater.cc:406] Registering TS with master... I0918 20:39:00.632750 23865 heartbeater.cc:455] Master 127.18.222.62:34677 requested a full tablet report, sending... I0918 20:39:00.633217 23683 ts_manager.cc:186] Registered new tserver with Master: 0b23c65819784c9d8689114664470d10 (127.18.222.1:43635) I0918 20:39:00.635639 23626 ts_manager.cc:186] Registered new tserver with Master: 0b23c65819784c9d8689114664470d10 (127.18.222.1:43635) I0918 20:39:00.636413 23683 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41052 I0918 20:39:00.640005 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 147592us I0918 20:39:00.640425 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:00.640601 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:00.640853 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:00.641027 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:00.642784 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/ts-1-root/instance: uuid: "37fb4dcf66be4d2298262038652468bd" format_stamp: "Formatted at 2019-09-18 20:39:00 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:00.649509 23864 heartbeater.cc:447] Master 127.18.222.61:33575 was elected leader, sending a full tablet report... I0918 20:39:00.654570 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.005s sys 0.008s I0918 20:39:00.656694 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:00.666221 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:00.666453 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/ts-1-root uuid: "37fb4dcf66be4d2298262038652468bd" format_stamp: "Formatted at 2019-09-18 20:39:00 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:00.666718 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:00.736974 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:00.737200 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:00.783414 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.2:37641 I0918 20:39:00.785318 19320 webserver.cc:374] Webserver started at http://127.18.222.2:45629/ using document root and password file I0918 20:39:00.814152 23934 heartbeater.cc:322] Connected to a master server at 127.18.222.60:32807 I0918 20:39:00.814558 23934 heartbeater.cc:406] Registering TS with master... I0918 20:39:00.815229 23934 heartbeater.cc:455] Master 127.18.222.60:32807 requested a full tablet report, sending... I0918 20:39:00.820333 23740 ts_manager.cc:186] Registered new tserver with Master: 37fb4dcf66be4d2298262038652468bd (127.18.222.2:37641) I0918 20:39:00.825951 23935 heartbeater.cc:322] Connected to a master server at 127.18.222.61:33575 I0918 20:39:00.826081 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 147592us I0918 20:39:00.826534 23935 heartbeater.cc:406] Registering TS with master... I0918 20:39:00.826843 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:00.827152 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:00.827450 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:00.827349 23935 heartbeater.cc:455] Master 127.18.222.61:33575 requested a full tablet report, sending... I0918 20:39:00.827770 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:00.828757 23937 heartbeater.cc:322] Connected to a master server at 127.18.222.62:34677 I0918 20:39:00.829083 23937 heartbeater.cc:406] Registering TS with master... I0918 20:39:00.829491 23937 heartbeater.cc:455] Master 127.18.222.62:34677 requested a full tablet report, sending... I0918 20:39:00.829413 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/ts-2-root/instance: uuid: "40e0e72158aa4ae6906b1da3389a0579" format_stamp: "Formatted at 2019-09-18 20:39:00 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:00.829737 23683 ts_manager.cc:186] Registered new tserver with Master: 37fb4dcf66be4d2298262038652468bd (127.18.222.2:37641) I0918 20:39:00.832056 23626 ts_manager.cc:186] Registered new tserver with Master: 37fb4dcf66be4d2298262038652468bd (127.18.222.2:37641) I0918 20:39:00.835440 23683 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41058 I0918 20:39:00.846065 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.016s user 0.007s sys 0.008s I0918 20:39:00.848948 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:00.851243 23935 heartbeater.cc:447] Master 127.18.222.61:33575 was elected leader, sending a full tablet report... I0918 20:39:00.859961 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:00.860352 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/ts-2-root uuid: "40e0e72158aa4ae6906b1da3389a0579" format_stamp: "Formatted at 2019-09-18 20:39:00 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:00.860795 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1568839089958813-19320/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:00.936472 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:00.936713 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:00.999162 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.3:34833 I0918 20:39:01.006009 19320 webserver.cc:374] Webserver started at http://127.18.222.3:37847/ using document root and password file I0918 20:39:01.052996 24007 heartbeater.cc:322] Connected to a master server at 127.18.222.62:34677 I0918 20:39:01.053401 24007 heartbeater.cc:406] Registering TS with master... I0918 20:39:01.054019 24007 heartbeater.cc:455] Master 127.18.222.62:34677 requested a full tablet report, sending... I0918 20:39:01.054848 24005 heartbeater.cc:322] Connected to a master server at 127.18.222.60:32807 I0918 20:39:01.055222 24005 heartbeater.cc:406] Registering TS with master... I0918 20:39:01.055752 24005 heartbeater.cc:455] Master 127.18.222.60:32807 requested a full tablet report, sending... I0918 20:39:01.056361 24006 heartbeater.cc:322] Connected to a master server at 127.18.222.61:33575 I0918 20:39:01.056643 24006 heartbeater.cc:406] Registering TS with master... I0918 20:39:01.057174 24006 heartbeater.cc:455] Master 127.18.222.61:33575 requested a full tablet report, sending... I0918 20:39:01.059164 23683 ts_manager.cc:186] Registered new tserver with Master: 40e0e72158aa4ae6906b1da3389a0579 (127.18.222.3:34833) I0918 20:39:01.059763 23625 ts_manager.cc:186] Registered new tserver with Master: 40e0e72158aa4ae6906b1da3389a0579 (127.18.222.3:34833) I0918 20:39:01.065299 23683 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41064 I0918 20:39:01.071166 23740 ts_manager.cc:186] Registered new tserver with Master: 40e0e72158aa4ae6906b1da3389a0579 (127.18.222.3:34833) I0918 20:39:01.071540 19320 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.047817438s I0918 20:39:01.087239 24006 heartbeater.cc:447] Master 127.18.222.61:33575 was elected leader, sending a full tablet report... I0918 20:39:01.158140 23683 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41070: name: "ksck-test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\n\000\000\000\004\001\024\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:39:01.160589 23683 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:39:01.279002 23830 tablet_service.cc:1022] Processing CreateTablet for tablet 237b2f5024fe4dd2b0741079640cc8fa (table=ksck-test-table [id=e501e79d09df41b7b7aad2cb7dbbcc92]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:01.280828 23829 tablet_service.cc:1022] Processing CreateTablet for tablet 40fb273a8bdf44438c2a534ad4d6850d (table=ksck-test-table [id=e501e79d09df41b7b7aad2cb7dbbcc92]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:01.281895 23829 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 40fb273a8bdf44438c2a534ad4d6850d. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:01.285043 23828 tablet_service.cc:1022] Processing CreateTablet for tablet 7a31e43eee9d4163a2f54129b325e20d (table=ksck-test-table [id=e501e79d09df41b7b7aad2cb7dbbcc92]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:01.286223 23828 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 7a31e43eee9d4163a2f54129b325e20d. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:01.288020 23830 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 237b2f5024fe4dd2b0741079640cc8fa. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:01.301767 23972 tablet_service.cc:1022] Processing CreateTablet for tablet 237b2f5024fe4dd2b0741079640cc8fa (table=ksck-test-table [id=e501e79d09df41b7b7aad2cb7dbbcc92]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:01.302976 23972 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 237b2f5024fe4dd2b0741079640cc8fa. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:01.304491 23971 tablet_service.cc:1022] Processing CreateTablet for tablet 40fb273a8bdf44438c2a534ad4d6850d (table=ksck-test-table [id=e501e79d09df41b7b7aad2cb7dbbcc92]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:01.306013 23970 tablet_service.cc:1022] Processing CreateTablet for tablet 7a31e43eee9d4163a2f54129b325e20d (table=ksck-test-table [id=e501e79d09df41b7b7aad2cb7dbbcc92]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:01.307970 23970 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 7a31e43eee9d4163a2f54129b325e20d. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:01.308898 23971 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 40fb273a8bdf44438c2a534ad4d6850d. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:01.340154 24034 tablet_bootstrap.cc:439] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10: Bootstrap starting. I0918 20:39:01.353080 24034 tablet_bootstrap.cc:590] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10: No blocks or log segments found. Creating new log. I0918 20:39:01.382459 23779 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59: acquiring CA information for follower catalog manager: success I0918 20:39:01.386572 23783 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0: acquiring CA information for follower catalog manager: success I0918 20:39:01.440913 23901 tablet_service.cc:1022] Processing CreateTablet for tablet 237b2f5024fe4dd2b0741079640cc8fa (table=ksck-test-table [id=e501e79d09df41b7b7aad2cb7dbbcc92]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:01.442905 23901 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 237b2f5024fe4dd2b0741079640cc8fa. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:01.443202 23899 tablet_service.cc:1022] Processing CreateTablet for tablet 7a31e43eee9d4163a2f54129b325e20d (table=ksck-test-table [id=e501e79d09df41b7b7aad2cb7dbbcc92]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:01.444340 23900 tablet_service.cc:1022] Processing CreateTablet for tablet 40fb273a8bdf44438c2a534ad4d6850d (table=ksck-test-table [id=e501e79d09df41b7b7aad2cb7dbbcc92]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:01.445310 23900 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 40fb273a8bdf44438c2a534ad4d6850d. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:01.446130 23783 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:01.448184 23899 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 7a31e43eee9d4163a2f54129b325e20d. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:01.462239 23779 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:01.476743 24036 tablet_bootstrap.cc:439] T 7a31e43eee9d4163a2f54129b325e20d P 40e0e72158aa4ae6906b1da3389a0579: Bootstrap starting. I0918 20:39:01.480692 24034 tablet_bootstrap.cc:439] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10: No bootstrap required, opened a new log I0918 20:39:01.481269 24034 ts_tablet_manager.cc:1085] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10: Time spent bootstrapping tablet: real 0.141s user 0.034s sys 0.054s I0918 20:39:01.483419 24034 raft_consensus.cc:340] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.485093 24034 raft_consensus.cc:366] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:01.485404 24034 raft_consensus.cc:705] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b23c65819784c9d8689114664470d10, State: Initialized, Role: FOLLOWER I0918 20:39:01.486176 24034 consensus_queue.cc:229] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.491009 24037 tablet_bootstrap.cc:439] T 237b2f5024fe4dd2b0741079640cc8fa P 37fb4dcf66be4d2298262038652468bd: Bootstrap starting. I0918 20:39:01.495165 24037 tablet_bootstrap.cc:590] T 237b2f5024fe4dd2b0741079640cc8fa P 37fb4dcf66be4d2298262038652468bd: No blocks or log segments found. Creating new log. I0918 20:39:01.504621 24034 ts_tablet_manager.cc:1113] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10: Time spent starting tablet: real 0.023s user 0.008s sys 0.004s I0918 20:39:01.505574 24034 tablet_bootstrap.cc:439] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10: Bootstrap starting. I0918 20:39:01.509541 24034 tablet_bootstrap.cc:590] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10: No blocks or log segments found. Creating new log. I0918 20:39:01.510730 24037 tablet_bootstrap.cc:439] T 237b2f5024fe4dd2b0741079640cc8fa P 37fb4dcf66be4d2298262038652468bd: No bootstrap required, opened a new log I0918 20:39:01.511252 24037 ts_tablet_manager.cc:1085] T 237b2f5024fe4dd2b0741079640cc8fa P 37fb4dcf66be4d2298262038652468bd: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.000s I0918 20:39:01.518139 24036 tablet_bootstrap.cc:590] T 7a31e43eee9d4163a2f54129b325e20d P 40e0e72158aa4ae6906b1da3389a0579: No blocks or log segments found. Creating new log. I0918 20:39:01.528080 24034 tablet_bootstrap.cc:439] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10: No bootstrap required, opened a new log I0918 20:39:01.523257 24037 raft_consensus.cc:340] T 237b2f5024fe4dd2b0741079640cc8fa P 37fb4dcf66be4d2298262038652468bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.529054 24037 raft_consensus.cc:366] T 237b2f5024fe4dd2b0741079640cc8fa P 37fb4dcf66be4d2298262038652468bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:01.529417 24037 raft_consensus.cc:705] T 237b2f5024fe4dd2b0741079640cc8fa P 37fb4dcf66be4d2298262038652468bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fb4dcf66be4d2298262038652468bd, State: Initialized, Role: FOLLOWER I0918 20:39:01.530671 24036 tablet_bootstrap.cc:439] T 7a31e43eee9d4163a2f54129b325e20d P 40e0e72158aa4ae6906b1da3389a0579: No bootstrap required, opened a new log I0918 20:39:01.530306 24037 consensus_queue.cc:229] T 237b2f5024fe4dd2b0741079640cc8fa P 37fb4dcf66be4d2298262038652468bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.531260 24036 ts_tablet_manager.cc:1085] T 7a31e43eee9d4163a2f54129b325e20d P 40e0e72158aa4ae6906b1da3389a0579: Time spent bootstrapping tablet: real 0.055s user 0.005s sys 0.003s I0918 20:39:01.529094 24034 ts_tablet_manager.cc:1085] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.000s I0918 20:39:01.536213 24036 raft_consensus.cc:340] T 7a31e43eee9d4163a2f54129b325e20d P 40e0e72158aa4ae6906b1da3389a0579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.536978 24036 raft_consensus.cc:366] T 7a31e43eee9d4163a2f54129b325e20d P 40e0e72158aa4ae6906b1da3389a0579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:01.537221 24036 raft_consensus.cc:705] T 7a31e43eee9d4163a2f54129b325e20d P 40e0e72158aa4ae6906b1da3389a0579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e0e72158aa4ae6906b1da3389a0579, State: Initialized, Role: FOLLOWER I0918 20:39:01.537621 24034 raft_consensus.cc:340] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.539052 24036 consensus_queue.cc:229] T 7a31e43eee9d4163a2f54129b325e20d P 40e0e72158aa4ae6906b1da3389a0579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.539343 24034 raft_consensus.cc:366] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:01.540762 24034 raft_consensus.cc:705] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b23c65819784c9d8689114664470d10, State: Initialized, Role: FOLLOWER I0918 20:39:01.546391 24037 ts_tablet_manager.cc:1113] T 237b2f5024fe4dd2b0741079640cc8fa P 37fb4dcf66be4d2298262038652468bd: Time spent starting tablet: real 0.035s user 0.006s sys 0.002s I0918 20:39:01.549000 24034 consensus_queue.cc:229] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.550593 24037 tablet_bootstrap.cc:439] T 7a31e43eee9d4163a2f54129b325e20d P 37fb4dcf66be4d2298262038652468bd: Bootstrap starting. I0918 20:39:01.556989 24036 ts_tablet_manager.cc:1113] T 7a31e43eee9d4163a2f54129b325e20d P 40e0e72158aa4ae6906b1da3389a0579: Time spent starting tablet: real 0.025s user 0.005s sys 0.001s I0918 20:39:01.560500 24036 tablet_bootstrap.cc:439] T 40fb273a8bdf44438c2a534ad4d6850d P 40e0e72158aa4ae6906b1da3389a0579: Bootstrap starting. I0918 20:39:01.574826 24034 ts_tablet_manager.cc:1113] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s I0918 20:39:01.578177 24036 tablet_bootstrap.cc:590] T 40fb273a8bdf44438c2a534ad4d6850d P 40e0e72158aa4ae6906b1da3389a0579: No blocks or log segments found. Creating new log. I0918 20:39:01.579946 24034 tablet_bootstrap.cc:439] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10: Bootstrap starting. I0918 20:39:01.583190 24037 tablet_bootstrap.cc:590] T 7a31e43eee9d4163a2f54129b325e20d P 37fb4dcf66be4d2298262038652468bd: No blocks or log segments found. Creating new log. I0918 20:39:01.586941 24034 tablet_bootstrap.cc:590] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10: No blocks or log segments found. Creating new log. I0918 20:39:01.599339 24037 tablet_bootstrap.cc:439] T 7a31e43eee9d4163a2f54129b325e20d P 37fb4dcf66be4d2298262038652468bd: No bootstrap required, opened a new log I0918 20:39:01.599882 24037 ts_tablet_manager.cc:1085] T 7a31e43eee9d4163a2f54129b325e20d P 37fb4dcf66be4d2298262038652468bd: Time spent bootstrapping tablet: real 0.053s user 0.007s sys 0.003s I0918 20:39:01.604200 24036 tablet_bootstrap.cc:439] T 40fb273a8bdf44438c2a534ad4d6850d P 40e0e72158aa4ae6906b1da3389a0579: No bootstrap required, opened a new log I0918 20:39:01.604845 24036 ts_tablet_manager.cc:1085] T 40fb273a8bdf44438c2a534ad4d6850d P 40e0e72158aa4ae6906b1da3389a0579: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.005s I0918 20:39:01.608520 24034 tablet_bootstrap.cc:439] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10: No bootstrap required, opened a new log I0918 20:39:01.609283 24034 ts_tablet_manager.cc:1085] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.000s I0918 20:39:01.602006 24037 raft_consensus.cc:340] T 7a31e43eee9d4163a2f54129b325e20d P 37fb4dcf66be4d2298262038652468bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.613029 24037 raft_consensus.cc:366] T 7a31e43eee9d4163a2f54129b325e20d P 37fb4dcf66be4d2298262038652468bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:01.613309 24037 raft_consensus.cc:705] T 7a31e43eee9d4163a2f54129b325e20d P 37fb4dcf66be4d2298262038652468bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fb4dcf66be4d2298262038652468bd, State: Initialized, Role: FOLLOWER I0918 20:39:01.615092 24034 raft_consensus.cc:340] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.615962 24034 raft_consensus.cc:366] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:01.616216 24034 raft_consensus.cc:705] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b23c65819784c9d8689114664470d10, State: Initialized, Role: FOLLOWER I0918 20:39:01.616828 24034 consensus_queue.cc:229] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.619045 24034 ts_tablet_manager.cc:1113] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s I0918 20:39:01.614758 24036 raft_consensus.cc:340] T 40fb273a8bdf44438c2a534ad4d6850d P 40e0e72158aa4ae6906b1da3389a0579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.629312 24036 raft_consensus.cc:366] T 40fb273a8bdf44438c2a534ad4d6850d P 40e0e72158aa4ae6906b1da3389a0579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:01.629593 24036 raft_consensus.cc:705] T 40fb273a8bdf44438c2a534ad4d6850d P 40e0e72158aa4ae6906b1da3389a0579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e0e72158aa4ae6906b1da3389a0579, State: Initialized, Role: FOLLOWER I0918 20:39:01.630318 24036 consensus_queue.cc:229] T 40fb273a8bdf44438c2a534ad4d6850d P 40e0e72158aa4ae6906b1da3389a0579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.631533 24037 consensus_queue.cc:229] T 7a31e43eee9d4163a2f54129b325e20d P 37fb4dcf66be4d2298262038652468bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.635459 24037 ts_tablet_manager.cc:1113] T 7a31e43eee9d4163a2f54129b325e20d P 37fb4dcf66be4d2298262038652468bd: Time spent starting tablet: real 0.035s user 0.005s sys 0.000s I0918 20:39:01.642238 24037 tablet_bootstrap.cc:439] T 40fb273a8bdf44438c2a534ad4d6850d P 37fb4dcf66be4d2298262038652468bd: Bootstrap starting. I0918 20:39:01.646440 24036 ts_tablet_manager.cc:1113] T 40fb273a8bdf44438c2a534ad4d6850d P 40e0e72158aa4ae6906b1da3389a0579: Time spent starting tablet: real 0.041s user 0.003s sys 0.002s I0918 20:39:01.652945 24036 tablet_bootstrap.cc:439] T 237b2f5024fe4dd2b0741079640cc8fa P 40e0e72158aa4ae6906b1da3389a0579: Bootstrap starting. I0918 20:39:01.656524 24036 tablet_bootstrap.cc:590] T 237b2f5024fe4dd2b0741079640cc8fa P 40e0e72158aa4ae6906b1da3389a0579: No blocks or log segments found. Creating new log. I0918 20:39:01.660630 24037 tablet_bootstrap.cc:590] T 40fb273a8bdf44438c2a534ad4d6850d P 37fb4dcf66be4d2298262038652468bd: No blocks or log segments found. Creating new log. I0918 20:39:01.669765 24036 tablet_bootstrap.cc:439] T 237b2f5024fe4dd2b0741079640cc8fa P 40e0e72158aa4ae6906b1da3389a0579: No bootstrap required, opened a new log I0918 20:39:01.670300 24036 ts_tablet_manager.cc:1085] T 237b2f5024fe4dd2b0741079640cc8fa P 40e0e72158aa4ae6906b1da3389a0579: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.009s I0918 20:39:01.677975 24036 raft_consensus.cc:340] T 237b2f5024fe4dd2b0741079640cc8fa P 40e0e72158aa4ae6906b1da3389a0579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.678759 24036 raft_consensus.cc:366] T 237b2f5024fe4dd2b0741079640cc8fa P 40e0e72158aa4ae6906b1da3389a0579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:01.678988 24036 raft_consensus.cc:705] T 237b2f5024fe4dd2b0741079640cc8fa P 40e0e72158aa4ae6906b1da3389a0579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e0e72158aa4ae6906b1da3389a0579, State: Initialized, Role: FOLLOWER I0918 20:39:01.679602 24036 consensus_queue.cc:229] T 237b2f5024fe4dd2b0741079640cc8fa P 40e0e72158aa4ae6906b1da3389a0579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.682096 24037 tablet_bootstrap.cc:439] T 40fb273a8bdf44438c2a534ad4d6850d P 37fb4dcf66be4d2298262038652468bd: No bootstrap required, opened a new log I0918 20:39:01.682780 24037 ts_tablet_manager.cc:1085] T 40fb273a8bdf44438c2a534ad4d6850d P 37fb4dcf66be4d2298262038652468bd: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.004s I0918 20:39:01.685261 24037 raft_consensus.cc:340] T 40fb273a8bdf44438c2a534ad4d6850d P 37fb4dcf66be4d2298262038652468bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.690989 24037 raft_consensus.cc:366] T 40fb273a8bdf44438c2a534ad4d6850d P 37fb4dcf66be4d2298262038652468bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:01.691299 24037 raft_consensus.cc:705] T 40fb273a8bdf44438c2a534ad4d6850d P 37fb4dcf66be4d2298262038652468bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fb4dcf66be4d2298262038652468bd, State: Initialized, Role: FOLLOWER I0918 20:39:01.693521 24037 consensus_queue.cc:229] T 40fb273a8bdf44438c2a534ad4d6850d P 37fb4dcf66be4d2298262038652468bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.697227 24037 ts_tablet_manager.cc:1113] T 40fb273a8bdf44438c2a534ad4d6850d P 37fb4dcf66be4d2298262038652468bd: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s I0918 20:39:01.698041 24036 ts_tablet_manager.cc:1113] T 237b2f5024fe4dd2b0741079640cc8fa P 40e0e72158aa4ae6906b1da3389a0579: Time spent starting tablet: real 0.027s user 0.003s sys 0.003s I0918 20:39:01.717967 24038 raft_consensus.cc:472] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:01.718591 24038 raft_consensus.cc:494] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.721599 24038 leader_election.cc:296] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37fb4dcf66be4d2298262038652468bd (127.18.222.2:37641), 40e0e72158aa4ae6906b1da3389a0579 (127.18.222.3:34833) I0918 20:39:01.747910 24038 raft_consensus.cc:472] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:01.750648 24038 raft_consensus.cc:494] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.754938 24038 leader_election.cc:296] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37fb4dcf66be4d2298262038652468bd (127.18.222.2:37641), 40e0e72158aa4ae6906b1da3389a0579 (127.18.222.3:34833) I0918 20:39:01.762859 23911 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "237b2f5024fe4dd2b0741079640cc8fa" candidate_uuid: "0b23c65819784c9d8689114664470d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fb4dcf66be4d2298262038652468bd" is_pre_election: true I0918 20:39:01.764047 23911 raft_consensus.cc:2350] T 237b2f5024fe4dd2b0741079640cc8fa P 37fb4dcf66be4d2298262038652468bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b23c65819784c9d8689114664470d10 in term 0. I0918 20:39:01.763417 23910 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "40fb273a8bdf44438c2a534ad4d6850d" candidate_uuid: "0b23c65819784c9d8689114664470d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fb4dcf66be4d2298262038652468bd" is_pre_election: true I0918 20:39:01.765619 23910 raft_consensus.cc:2350] T 40fb273a8bdf44438c2a534ad4d6850d P 37fb4dcf66be4d2298262038652468bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b23c65819784c9d8689114664470d10 in term 0. I0918 20:39:01.767372 23789 leader_election.cc:310] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b23c65819784c9d8689114664470d10, 37fb4dcf66be4d2298262038652468bd; no voters: I0918 20:39:01.768384 24038 raft_consensus.cc:2644] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:01.768524 23789 leader_election.cc:310] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b23c65819784c9d8689114664470d10, 37fb4dcf66be4d2298262038652468bd; no voters: I0918 20:39:01.769493 24038 raft_consensus.cc:472] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:01.769834 24038 raft_consensus.cc:2890] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:01.779875 24038 raft_consensus.cc:494] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.781255 24038 leader_election.cc:296] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10 [CANDIDATE]: Term 1 election: Requested vote from peers 37fb4dcf66be4d2298262038652468bd (127.18.222.2:37641), 40e0e72158aa4ae6906b1da3389a0579 (127.18.222.3:34833) I0918 20:39:01.781729 24038 raft_consensus.cc:2644] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:01.781958 24038 raft_consensus.cc:472] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:01.782196 24038 raft_consensus.cc:2890] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:01.788601 23910 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "237b2f5024fe4dd2b0741079640cc8fa" candidate_uuid: "0b23c65819784c9d8689114664470d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fb4dcf66be4d2298262038652468bd" I0918 20:39:01.789362 23910 raft_consensus.cc:2890] T 237b2f5024fe4dd2b0741079640cc8fa P 37fb4dcf66be4d2298262038652468bd [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:01.789131 24038 raft_consensus.cc:494] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.790539 24038 leader_election.cc:296] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10 [CANDIDATE]: Term 1 election: Requested vote from peers 37fb4dcf66be4d2298262038652468bd (127.18.222.2:37641), 40e0e72158aa4ae6906b1da3389a0579 (127.18.222.3:34833) I0918 20:39:01.794111 23910 raft_consensus.cc:2350] T 237b2f5024fe4dd2b0741079640cc8fa P 37fb4dcf66be4d2298262038652468bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b23c65819784c9d8689114664470d10 in term 1. I0918 20:39:01.796833 23982 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "237b2f5024fe4dd2b0741079640cc8fa" candidate_uuid: "0b23c65819784c9d8689114664470d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e0e72158aa4ae6906b1da3389a0579" is_pre_election: true I0918 20:39:01.797536 23982 raft_consensus.cc:2350] T 237b2f5024fe4dd2b0741079640cc8fa P 40e0e72158aa4ae6906b1da3389a0579 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b23c65819784c9d8689114664470d10 in term 0. I0918 20:39:01.804127 23980 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "40fb273a8bdf44438c2a534ad4d6850d" candidate_uuid: "0b23c65819784c9d8689114664470d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e0e72158aa4ae6906b1da3389a0579" I0918 20:39:01.804705 23980 raft_consensus.cc:2890] T 40fb273a8bdf44438c2a534ad4d6850d P 40e0e72158aa4ae6906b1da3389a0579 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:01.805670 23789 leader_election.cc:310] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b23c65819784c9d8689114664470d10, 37fb4dcf66be4d2298262038652468bd; no voters: I0918 20:39:01.806923 24038 raft_consensus.cc:2644] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:01.807390 24038 raft_consensus.cc:667] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10 [term 1 LEADER]: Becoming Leader. State: Replica: 0b23c65819784c9d8689114664470d10, State: Running, Role: LEADER I0918 20:39:01.803458 23981 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "237b2f5024fe4dd2b0741079640cc8fa" candidate_uuid: "0b23c65819784c9d8689114664470d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e0e72158aa4ae6906b1da3389a0579" I0918 20:39:01.808140 23981 raft_consensus.cc:2890] T 237b2f5024fe4dd2b0741079640cc8fa P 40e0e72158aa4ae6906b1da3389a0579 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:01.808051 24038 consensus_queue.cc:206] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.809250 23982 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "40fb273a8bdf44438c2a534ad4d6850d" candidate_uuid: "0b23c65819784c9d8689114664470d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e0e72158aa4ae6906b1da3389a0579" is_pre_election: true I0918 20:39:01.810760 23910 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "40fb273a8bdf44438c2a534ad4d6850d" candidate_uuid: "0b23c65819784c9d8689114664470d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fb4dcf66be4d2298262038652468bd" I0918 20:39:01.811393 23910 raft_consensus.cc:2890] T 40fb273a8bdf44438c2a534ad4d6850d P 37fb4dcf66be4d2298262038652468bd [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:01.817570 23980 raft_consensus.cc:2350] T 40fb273a8bdf44438c2a534ad4d6850d P 40e0e72158aa4ae6906b1da3389a0579 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b23c65819784c9d8689114664470d10 in term 1. I0918 20:39:01.818264 23982 raft_consensus.cc:2325] T 40fb273a8bdf44438c2a534ad4d6850d P 40e0e72158aa4ae6906b1da3389a0579 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0b23c65819784c9d8689114664470d10 for term 1 because replica is already servicing an update from a current leader or another vote. I0918 20:39:01.822737 23981 raft_consensus.cc:2350] T 237b2f5024fe4dd2b0741079640cc8fa P 40e0e72158aa4ae6906b1da3389a0579 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b23c65819784c9d8689114664470d10 in term 1. I0918 20:39:01.823372 23910 raft_consensus.cc:2350] T 40fb273a8bdf44438c2a534ad4d6850d P 37fb4dcf66be4d2298262038652468bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b23c65819784c9d8689114664470d10 in term 1. I0918 20:39:01.824993 23789 leader_election.cc:310] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b23c65819784c9d8689114664470d10, 37fb4dcf66be4d2298262038652468bd; no voters: I0918 20:39:01.825999 24051 raft_consensus.cc:2644] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:01.827900 24051 raft_consensus.cc:667] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10 [term 1 LEADER]: Becoming Leader. State: Replica: 0b23c65819784c9d8689114664470d10, State: Running, Role: LEADER I0918 20:39:01.828734 24051 consensus_queue.cc:206] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.840075 23682 catalog_manager.cc:4131] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10 reported cstate change: term changed from 0 to 1, leader changed from to 0b23c65819784c9d8689114664470d10 (127.18.222.1). New cstate: current_term: 1 leader_uuid: "0b23c65819784c9d8689114664470d10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } health_report { overall_health: UNKNOWN } } } I0918 20:39:01.877054 23680 catalog_manager.cc:4131] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10 reported cstate change: term changed from 0 to 1, leader changed from to 0b23c65819784c9d8689114664470d10 (127.18.222.1). New cstate: current_term: 1 leader_uuid: "0b23c65819784c9d8689114664470d10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } health_report { overall_health: UNKNOWN } } } I0918 20:39:01.938084 24051 raft_consensus.cc:472] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:01.938534 24051 raft_consensus.cc:494] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.939915 24051 leader_election.cc:296] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37fb4dcf66be4d2298262038652468bd (127.18.222.2:37641), 40e0e72158aa4ae6906b1da3389a0579 (127.18.222.3:34833) I0918 20:39:01.940773 23910 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7a31e43eee9d4163a2f54129b325e20d" candidate_uuid: "0b23c65819784c9d8689114664470d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fb4dcf66be4d2298262038652468bd" is_pre_election: true I0918 20:39:01.941018 23981 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7a31e43eee9d4163a2f54129b325e20d" candidate_uuid: "0b23c65819784c9d8689114664470d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e0e72158aa4ae6906b1da3389a0579" is_pre_election: true I0918 20:39:01.941505 23910 raft_consensus.cc:2350] T 7a31e43eee9d4163a2f54129b325e20d P 37fb4dcf66be4d2298262038652468bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b23c65819784c9d8689114664470d10 in term 0. I0918 20:39:01.941690 23981 raft_consensus.cc:2350] T 7a31e43eee9d4163a2f54129b325e20d P 40e0e72158aa4ae6906b1da3389a0579 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b23c65819784c9d8689114664470d10 in term 0. I0918 20:39:01.942692 23789 leader_election.cc:310] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b23c65819784c9d8689114664470d10, 37fb4dcf66be4d2298262038652468bd; no voters: I0918 20:39:01.943526 24051 raft_consensus.cc:2644] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:01.943914 24051 raft_consensus.cc:472] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:01.944221 24051 raft_consensus.cc:2890] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:01.953446 24051 raft_consensus.cc:494] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.955003 24051 leader_election.cc:296] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10 [CANDIDATE]: Term 1 election: Requested vote from peers 37fb4dcf66be4d2298262038652468bd (127.18.222.2:37641), 40e0e72158aa4ae6906b1da3389a0579 (127.18.222.3:34833) I0918 20:39:01.956109 23981 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7a31e43eee9d4163a2f54129b325e20d" candidate_uuid: "0b23c65819784c9d8689114664470d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e0e72158aa4ae6906b1da3389a0579" I0918 20:39:01.956109 23910 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7a31e43eee9d4163a2f54129b325e20d" candidate_uuid: "0b23c65819784c9d8689114664470d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fb4dcf66be4d2298262038652468bd" I0918 20:39:01.956938 23981 raft_consensus.cc:2890] T 7a31e43eee9d4163a2f54129b325e20d P 40e0e72158aa4ae6906b1da3389a0579 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:01.957058 23910 raft_consensus.cc:2890] T 7a31e43eee9d4163a2f54129b325e20d P 37fb4dcf66be4d2298262038652468bd [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:01.962994 23981 raft_consensus.cc:2350] T 7a31e43eee9d4163a2f54129b325e20d P 40e0e72158aa4ae6906b1da3389a0579 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b23c65819784c9d8689114664470d10 in term 1. I0918 20:39:01.965919 23789 leader_election.cc:310] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b23c65819784c9d8689114664470d10, 40e0e72158aa4ae6906b1da3389a0579; no voters: I0918 20:39:01.966884 24051 raft_consensus.cc:2644] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:01.967782 24051 raft_consensus.cc:667] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10 [term 1 LEADER]: Becoming Leader. State: Replica: 0b23c65819784c9d8689114664470d10, State: Running, Role: LEADER I0918 20:39:01.967478 23910 raft_consensus.cc:2350] T 7a31e43eee9d4163a2f54129b325e20d P 37fb4dcf66be4d2298262038652468bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b23c65819784c9d8689114664470d10 in term 1. I0918 20:39:01.968668 24051 consensus_queue.cc:206] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } } I0918 20:39:01.974761 23681 catalog_manager.cc:4131] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10 reported cstate change: term changed from 0 to 1, leader changed from to 0b23c65819784c9d8689114664470d10 (127.18.222.1). New cstate: current_term: 1 leader_uuid: "0b23c65819784c9d8689114664470d10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b23c65819784c9d8689114664470d10" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 43635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37fb4dcf66be4d2298262038652468bd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40e0e72158aa4ae6906b1da3389a0579" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34833 } health_report { overall_health: UNKNOWN } } } W0918 20:39:02.039639 19320 ksck_remote-test.cc:521] test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run I0918 20:39:02.040050 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:02.126307 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:02.126921 19320 tablet_replica.cc:273] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10: stopping tablet replica I0918 20:39:02.127549 19320 raft_consensus.cc:2147] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:02.128337 19320 pending_rounds.cc:64] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10: Trying to abort 1 pending transactions. I0918 20:39:02.128542 19320 pending_rounds.cc:70] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6425965124858384384 op_type: NO_OP noop_request { } I0918 20:39:02.128895 19320 raft_consensus.cc:2727] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted I0918 20:39:02.129117 19320 raft_consensus.cc:2174] T 237b2f5024fe4dd2b0741079640cc8fa P 0b23c65819784c9d8689114664470d10 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:02.132473 19320 tablet_replica.cc:273] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10: stopping tablet replica I0918 20:39:02.133061 19320 raft_consensus.cc:2147] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:02.133703 19320 pending_rounds.cc:64] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10: Trying to abort 1 pending transactions. I0918 20:39:02.133898 19320 pending_rounds.cc:70] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6425965125510983680 op_type: NO_OP noop_request { } I0918 20:39:02.134246 19320 raft_consensus.cc:2727] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted I0918 20:39:02.134516 19320 raft_consensus.cc:2174] T 7a31e43eee9d4163a2f54129b325e20d P 0b23c65819784c9d8689114664470d10 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:02.137059 19320 tablet_replica.cc:273] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10: stopping tablet replica I0918 20:39:02.137429 19320 raft_consensus.cc:2147] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:02.137889 19320 pending_rounds.cc:64] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10: Trying to abort 1 pending transactions. I0918 20:39:02.138033 19320 pending_rounds.cc:70] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6425965124938510336 op_type: NO_OP noop_request { } I0918 20:39:02.138305 19320 raft_consensus.cc:2727] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted I0918 20:39:02.138523 19320 raft_consensus.cc:2174] T 40fb273a8bdf44438c2a534ad4d6850d P 0b23c65819784c9d8689114664470d10 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:02.165500 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:02.165724 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:02.166079 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:02.182662 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:02.225623 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:02.225960 19320 tablet_replica.cc:273] T 40fb273a8bdf44438c2a534ad4d6850d P 37fb4dcf66be4d2298262038652468bd: stopping tablet replica I0918 20:39:02.226341 19320 raft_consensus.cc:2147] T 40fb273a8bdf44438c2a534ad4d6850d P 37fb4dcf66be4d2298262038652468bd [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:02.226625 19320 raft_consensus.cc:2174] T 40fb273a8bdf44438c2a534ad4d6850d P 37fb4dcf66be4d2298262038652468bd [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:02.228332 19320 tablet_replica.cc:273] T 7a31e43eee9d4163a2f54129b325e20d P 37fb4dcf66be4d2298262038652468bd: stopping tablet replica I0918 20:39:02.228669 19320 raft_consensus.cc:2147] T 7a31e43eee9d4163a2f54129b325e20d P 37fb4dcf66be4d2298262038652468bd [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:02.228951 19320 raft_consensus.cc:2174] T 7a31e43eee9d4163a2f54129b325e20d P 37fb4dcf66be4d2298262038652468bd [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:02.230360 19320 tablet_replica.cc:273] T 237b2f5024fe4dd2b0741079640cc8fa P 37fb4dcf66be4d2298262038652468bd: stopping tablet replica I0918 20:39:02.230666 19320 raft_consensus.cc:2147] T 237b2f5024fe4dd2b0741079640cc8fa P 37fb4dcf66be4d2298262038652468bd [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:02.230931 19320 raft_consensus.cc:2174] T 237b2f5024fe4dd2b0741079640cc8fa P 37fb4dcf66be4d2298262038652468bd [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:02.254037 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:02.254225 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:02.254534 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:02.264653 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:02.301908 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:02.302284 19320 tablet_replica.cc:273] T 237b2f5024fe4dd2b0741079640cc8fa P 40e0e72158aa4ae6906b1da3389a0579: stopping tablet replica I0918 20:39:02.302655 19320 raft_consensus.cc:2147] T 237b2f5024fe4dd2b0741079640cc8fa P 40e0e72158aa4ae6906b1da3389a0579 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:02.302949 19320 raft_consensus.cc:2174] T 237b2f5024fe4dd2b0741079640cc8fa P 40e0e72158aa4ae6906b1da3389a0579 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:02.304651 19320 tablet_replica.cc:273] T 40fb273a8bdf44438c2a534ad4d6850d P 40e0e72158aa4ae6906b1da3389a0579: stopping tablet replica I0918 20:39:02.305017 19320 raft_consensus.cc:2147] T 40fb273a8bdf44438c2a534ad4d6850d P 40e0e72158aa4ae6906b1da3389a0579 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:02.305282 19320 raft_consensus.cc:2174] T 40fb273a8bdf44438c2a534ad4d6850d P 40e0e72158aa4ae6906b1da3389a0579 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:02.306651 19320 tablet_replica.cc:273] T 7a31e43eee9d4163a2f54129b325e20d P 40e0e72158aa4ae6906b1da3389a0579: stopping tablet replica I0918 20:39:02.306964 19320 raft_consensus.cc:2147] T 7a31e43eee9d4163a2f54129b325e20d P 40e0e72158aa4ae6906b1da3389a0579 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:02.307211 19320 raft_consensus.cc:2174] T 7a31e43eee9d4163a2f54129b325e20d P 40e0e72158aa4ae6906b1da3389a0579 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:02.327383 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:02.327584 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:02.327984 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:02.337442 19320 master.cc:247] Master@127.18.222.62:34677 shutting down... I0918 20:39:02.358021 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:02.358430 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:02.358677 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 9677e12125ca44d483fd9cab2182a2a0: stopping tablet replica I0918 20:39:02.378521 19320 master.cc:258] Master@127.18.222.62:34677 shutdown complete. I0918 20:39:02.390614 19320 master.cc:247] Master@127.18.222.61:33575 shutting down... I0918 20:39:02.409600 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:02.410360 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:02.410594 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 5149d1f41d644da7ad63a91236c71d5b: stopping tablet replica I0918 20:39:02.431217 19320 master.cc:258] Master@127.18.222.61:33575 shutdown complete. I0918 20:39:02.441004 19320 master.cc:247] Master@127.18.222.60:32807 shutting down... I0918 20:39:02.460402 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:02.460777 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:02.461055 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 120945e9b78f41feaa64417f83cdee59: stopping tablet replica I0918 20:39:02.479689 19320 master.cc:258] Master@127.18.222.60:32807 shutdown complete. [ OK ] RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM (2560 ms) [ RUN ] RemoteKsckTest.TestLeaderMasterDown I0918 20:39:02.512478 19320 test_util.cc:212] Using random seed: -741566655 I0918 20:39:02.513278 19320 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.18.222.62:43565,127.18.222.61:39201,127.18.222.60:46245 I0918 20:39:02.525564 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 148592us I0918 20:39:02.525949 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:02.526108 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:02.526340 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:02.526470 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:02.527509 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/master-0-root/instance: uuid: "17f33eef4f4c4000a2dc645ff32512d7" format_stamp: "Formatted at 2019-09-18 20:39:02 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:02.536103 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s I0918 20:39:02.537655 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:02.544682 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:02.544901 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/master-0-root uuid: "17f33eef4f4c4000a2dc645ff32512d7" format_stamp: "Formatted at 2019-09-18 20:39:02 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:02.545157 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:02.679890 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:43565 I0918 20:39:02.681933 19320 webserver.cc:374] Webserver started at http://127.18.222.62:44613/ using document root and password file I0918 20:39:02.684264 24122 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:02.691895 24122 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:02.704277 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 148592us I0918 20:39:02.704723 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:02.704979 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:02.705250 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:02.705471 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:02.706714 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/master-1-root/instance: uuid: "b69dbcabcdda4e728f542c193ca7b1b2" format_stamp: "Formatted at 2019-09-18 20:39:02 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:02.707044 24122 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:02.709281 24122 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:39201: Network error: Client connection negotiation failed: client connection to 127.18.222.61:39201: connect: Connection refused (error 111) I0918 20:39:02.717209 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.011s sys 0.000s I0918 20:39:02.718895 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:02.728225 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0918 20:39:02.728438 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/master-1-root uuid: "b69dbcabcdda4e728f542c193ca7b1b2" format_stamp: "Formatted at 2019-09-18 20:39:02 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:02.728695 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:02.752669 24122 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } attempt: 1 W0918 20:39:02.754984 24122 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:39201: Network error: Client connection negotiation failed: client connection to 127.18.222.61:39201: connect: Connection refused (error 111) I0918 20:39:02.789688 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:39201 I0918 20:39:02.791679 19320 webserver.cc:374] Webserver started at http://127.18.222.61:46531/ using document root and password file I0918 20:39:02.793514 24179 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:02.798480 24179 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:02.809814 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 148592us I0918 20:39:02.809470 24122 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } attempt: 2 I0918 20:39:02.810271 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:02.810518 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:02.810902 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:02.811146 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:02.812594 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/master-2-root/instance: uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" format_stamp: "Formatted at 2019-09-18 20:39:02 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:02.815606 24179 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:02.827333 24122 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:02.829300 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.016s user 0.016s sys 0.000s W0918 20:39:02.829512 24122 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:46245: Network error: Client connection negotiation failed: client connection to 127.18.222.60:46245: connect: Connection refused (error 111) I0918 20:39:02.831307 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:02.831446 24179 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:02.833215 24179 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:46245: Network error: Client connection negotiation failed: client connection to 127.18.222.60:46245: connect: Connection refused (error 111) I0918 20:39:02.840095 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:39:02.840407 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/master-2-root uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" format_stamp: "Formatted at 2019-09-18 20:39:02 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:02.840669 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:02.890264 24122 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } attempt: 1 I0918 20:39:02.890517 24179 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } attempt: 1 W0918 20:39:02.893043 24179 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:46245: Network error: Client connection negotiation failed: client connection to 127.18.222.60:46245: connect: Connection refused (error 111) W0918 20:39:02.893482 24122 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:46245: Network error: Client connection negotiation failed: client connection to 127.18.222.60:46245: connect: Connection refused (error 111) I0918 20:39:02.901348 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:46245 I0918 20:39:02.903193 19320 webserver.cc:374] Webserver started at http://127.18.222.60:39999/ using document root and password file I0918 20:39:02.904937 24237 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:02.907799 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0918 20:39:02.910009 24237 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:02.920855 24237 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:02.930510 24237 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:02.935967 24122 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } attempt: 2 I0918 20:39:02.945492 24179 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } attempt: 2 I0918 20:39:02.948336 24237 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc: Bootstrap starting. I0918 20:39:02.952308 24237 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc: No blocks or log segments found. Creating new log. I0918 20:39:02.953990 24122 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7: Bootstrap starting. I0918 20:39:02.955893 24237 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc: No bootstrap required, opened a new log I0918 20:39:02.957696 24122 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7: No blocks or log segments found. Creating new log. I0918 20:39:02.958221 24237 raft_consensus.cc:340] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } I0918 20:39:02.959061 24237 raft_consensus.cc:366] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:02.959298 24237 raft_consensus.cc:705] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fe1f99a02a849dca2a9e9aa4ed858dc, State: Initialized, Role: FOLLOWER I0918 20:39:02.960041 24237 consensus_queue.cc:229] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } I0918 20:39:02.967766 24179 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2: Bootstrap starting. I0918 20:39:02.972543 24237 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } } I0918 20:39:02.972807 24243 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } } I0918 20:39:02.973109 24179 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2: No blocks or log segments found. Creating new log. I0918 20:39:02.973649 24243 sys_catalog.cc:343] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:02.973891 24122 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7: No bootstrap required, opened a new log I0918 20:39:02.973513 24237 sys_catalog.cc:343] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:02.976101 24237 sys_catalog.cc:428] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:02.975972 24122 raft_consensus.cc:340] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } I0918 20:39:02.976536 24122 raft_consensus.cc:366] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:02.976729 24122 raft_consensus.cc:705] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f33eef4f4c4000a2dc645ff32512d7, State: Initialized, Role: FOLLOWER I0918 20:39:02.977406 24122 consensus_queue.cc:229] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } I0918 20:39:02.978601 24179 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2: No bootstrap required, opened a new log W0918 20:39:02.980542 24245 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc: acquiring CA information for follower catalog manager: Not found: root CA entry not found I0918 20:39:02.980370 24122 sys_catalog.cc:340] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } } W0918 20:39:02.980943 24245 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:02.980659 24246 sys_catalog.cc:340] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } } I0918 20:39:02.981298 24122 sys_catalog.cc:343] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:02.981513 24246 sys_catalog.cc:343] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:02.981667 24179 raft_consensus.cc:340] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } I0918 20:39:02.982391 24122 sys_catalog.cc:428] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:02.982403 24179 raft_consensus.cc:366] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:02.982759 24179 raft_consensus.cc:705] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69dbcabcdda4e728f542c193ca7b1b2, State: Initialized, Role: FOLLOWER I0918 20:39:02.984174 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 W0918 20:39:02.986110 24247 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:02.986392 24247 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:02.983300 24179 consensus_queue.cc:229] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } I0918 20:39:02.987839 24248 sys_catalog.cc:340] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } } I0918 20:39:02.988378 24248 sys_catalog.cc:343] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:02.988813 24179 sys_catalog.cc:340] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } } I0918 20:39:02.989279 24179 sys_catalog.cc:343] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:02.990180 24179 sys_catalog.cc:428] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:02.991142 24249 catalog_manager.cc:569] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0918 20:39:02.991521 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 I0918 20:39:03.007292 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 149092us I0918 20:39:03.007767 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:03.007953 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:03.008186 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:03.008411 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:03.009631 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/ts-0-root/instance: uuid: "18903fadbbe345ee98fd47291e7835f4" format_stamp: "Formatted at 2019-09-18 20:39:03 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:03.020520 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.012s sys 0.000s I0918 20:39:03.022284 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:03.030640 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:39:03.030887 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/ts-0-root uuid: "18903fadbbe345ee98fd47291e7835f4" format_stamp: "Formatted at 2019-09-18 20:39:03 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:03.031160 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:03.059985 24243 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:03.060572 24243 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } I0918 20:39:03.062345 24243 leader_election.cc:296] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17f33eef4f4c4000a2dc645ff32512d7 (127.18.222.62:43565), b69dbcabcdda4e728f542c193ca7b1b2 (127.18.222.61:39201) I0918 20:39:03.063331 24156 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" is_pre_election: true I0918 20:39:03.063285 24099 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17f33eef4f4c4000a2dc645ff32512d7" is_pre_election: true I0918 20:39:03.064198 24156 raft_consensus.cc:2350] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fe1f99a02a849dca2a9e9aa4ed858dc in term 0. I0918 20:39:03.064229 24099 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fe1f99a02a849dca2a9e9aa4ed858dc in term 0. I0918 20:39:03.065301 24185 leader_election.cc:310] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fe1f99a02a849dca2a9e9aa4ed858dc, b69dbcabcdda4e728f542c193ca7b1b2; no voters: I0918 20:39:03.065981 24243 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:03.066280 24243 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:03.066557 24243 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:03.074329 24243 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } I0918 20:39:03.076853 24243 leader_election.cc:296] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [CANDIDATE]: Term 1 election: Requested vote from peers 17f33eef4f4c4000a2dc645ff32512d7 (127.18.222.62:43565), b69dbcabcdda4e728f542c193ca7b1b2 (127.18.222.61:39201) I0918 20:39:03.077895 24156 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" I0918 20:39:03.077786 24099 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17f33eef4f4c4000a2dc645ff32512d7" I0918 20:39:03.078574 24156 raft_consensus.cc:2890] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:03.078723 24099 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:03.083251 24156 raft_consensus.cc:2350] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fe1f99a02a849dca2a9e9aa4ed858dc in term 1. I0918 20:39:03.084601 24185 leader_election.cc:310] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fe1f99a02a849dca2a9e9aa4ed858dc, b69dbcabcdda4e728f542c193ca7b1b2; no voters: I0918 20:39:03.085435 24243 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:03.086321 24099 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fe1f99a02a849dca2a9e9aa4ed858dc in term 1. I0918 20:39:03.088570 24243 raft_consensus.cc:667] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [term 1 LEADER]: Becoming Leader. State: Replica: 2fe1f99a02a849dca2a9e9aa4ed858dc, State: Running, Role: LEADER I0918 20:39:03.089114 24243 consensus_queue.cc:206] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } I0918 20:39:03.091164 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:03.091428 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:03.092998 24259 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fe1f99a02a849dca2a9e9aa4ed858dc. Latest consensus state: current_term: 1 leader_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } } I0918 20:39:03.093875 24259 sys_catalog.cc:343] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [sys.catalog]: This master's current role is: LEADER I0918 20:39:03.095644 24263 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:39:03.098281 24263 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:39:03.118904 24099 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [term 1 FOLLOWER]: Refusing update from remote peer 2fe1f99a02a849dca2a9e9aa4ed858dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:03.118897 24156 raft_consensus.cc:1184] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [term 1 FOLLOWER]: Refusing update from remote peer 2fe1f99a02a849dca2a9e9aa4ed858dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:03.120981 24259 consensus_queue.cc:984] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:03.121831 24243 consensus_queue.cc:984] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:03.130622 24248 sys_catalog.cc:340] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fe1f99a02a849dca2a9e9aa4ed858dc. Latest consensus state: current_term: 1 leader_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } } I0918 20:39:03.131891 24248 sys_catalog.cc:343] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:03.134793 24246 sys_catalog.cc:340] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fe1f99a02a849dca2a9e9aa4ed858dc. Latest consensus state: current_term: 1 leader_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } } I0918 20:39:03.135874 24246 sys_catalog.cc:343] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:03.143514 24246 sys_catalog.cc:340] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } } I0918 20:39:03.144196 24246 sys_catalog.cc:343] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:03.147130 24259 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } } I0918 20:39:03.148533 24259 sys_catalog.cc:343] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [sys.catalog]: This master's current role is: LEADER I0918 20:39:03.150110 24259 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } } I0918 20:39:03.150455 24248 sys_catalog.cc:340] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } } I0918 20:39:03.150915 24259 sys_catalog.cc:343] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [sys.catalog]: This master's current role is: LEADER I0918 20:39:03.151115 24248 sys_catalog.cc:343] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:03.152415 24263 catalog_manager.cc:981] Generated new certificate authority record I0918 20:39:03.154390 24263 catalog_manager.cc:1106] Loading token signing keys... I0918 20:39:03.175873 24263 catalog_manager.cc:4446] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc: Generated new TSK 0 I0918 20:39:03.176712 24263 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:39:03.200457 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:33561 I0918 20:39:03.202644 19320 webserver.cc:374] Webserver started at http://127.18.222.1:39753/ using document root and password file I0918 20:39:03.229380 24328 heartbeater.cc:322] Connected to a master server at 127.18.222.62:43565 I0918 20:39:03.229887 24328 heartbeater.cc:406] Registering TS with master... I0918 20:39:03.229898 24326 heartbeater.cc:322] Connected to a master server at 127.18.222.60:46245 I0918 20:39:03.230340 24326 heartbeater.cc:406] Registering TS with master... I0918 20:39:03.230593 24328 heartbeater.cc:455] Master 127.18.222.62:43565 requested a full tablet report, sending... I0918 20:39:03.230901 24327 heartbeater.cc:322] Connected to a master server at 127.18.222.61:39201 I0918 20:39:03.230870 24326 heartbeater.cc:455] Master 127.18.222.60:46245 requested a full tablet report, sending... I0918 20:39:03.231277 24327 heartbeater.cc:406] Registering TS with master... I0918 20:39:03.232012 24327 heartbeater.cc:455] Master 127.18.222.61:39201 requested a full tablet report, sending... I0918 20:39:03.233202 24204 ts_manager.cc:186] Registered new tserver with Master: 18903fadbbe345ee98fd47291e7835f4 (127.18.222.1:33561) I0918 20:39:03.233237 24089 ts_manager.cc:186] Registered new tserver with Master: 18903fadbbe345ee98fd47291e7835f4 (127.18.222.1:33561) I0918 20:39:03.234625 24146 ts_manager.cc:186] Registered new tserver with Master: 18903fadbbe345ee98fd47291e7835f4 (127.18.222.1:33561) I0918 20:39:03.236435 24204 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49438 I0918 20:39:03.238514 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 149092us I0918 20:39:03.238893 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:03.239064 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:03.239284 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:03.239419 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:03.240459 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/ts-1-root/instance: uuid: "9ac9d6c4caff4b8dab06322af4fda065" format_stamp: "Formatted at 2019-09-18 20:39:03 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:03.249080 24326 heartbeater.cc:447] Master 127.18.222.60:46245 was elected leader, sending a full tablet report... I0918 20:39:03.253767 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.013s user 0.011s sys 0.002s I0918 20:39:03.255421 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:03.265393 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:03.265605 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/ts-1-root uuid: "9ac9d6c4caff4b8dab06322af4fda065" format_stamp: "Formatted at 2019-09-18 20:39:03 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:03.265866 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:03.290493 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:03.290689 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:03.356256 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.2:40761 I0918 20:39:03.358031 19320 webserver.cc:374] Webserver started at http://127.18.222.2:42399/ using document root and password file I0918 20:39:03.391819 24399 heartbeater.cc:322] Connected to a master server at 127.18.222.62:43565 I0918 20:39:03.392185 24399 heartbeater.cc:406] Registering TS with master... I0918 20:39:03.392858 24399 heartbeater.cc:455] Master 127.18.222.62:43565 requested a full tablet report, sending... I0918 20:39:03.395282 24089 ts_manager.cc:186] Registered new tserver with Master: 9ac9d6c4caff4b8dab06322af4fda065 (127.18.222.2:40761) I0918 20:39:03.397136 24397 heartbeater.cc:322] Connected to a master server at 127.18.222.60:46245 I0918 20:39:03.397627 24397 heartbeater.cc:406] Registering TS with master... I0918 20:39:03.398313 24397 heartbeater.cc:455] Master 127.18.222.60:46245 requested a full tablet report, sending... I0918 20:39:03.400172 24204 ts_manager.cc:186] Registered new tserver with Master: 9ac9d6c4caff4b8dab06322af4fda065 (127.18.222.2:40761) I0918 20:39:03.401028 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 149092us I0918 20:39:03.401373 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:03.401546 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:03.401859 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:03.402012 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:03.402458 24204 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49444 I0918 20:39:03.403225 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/ts-2-root/instance: uuid: "18498189d0874c39adf8a1bcd497997a" format_stamp: "Formatted at 2019-09-18 20:39:03 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:03.404500 24398 heartbeater.cc:322] Connected to a master server at 127.18.222.61:39201 I0918 20:39:03.404901 24398 heartbeater.cc:406] Registering TS with master... I0918 20:39:03.405529 24398 heartbeater.cc:455] Master 127.18.222.61:39201 requested a full tablet report, sending... I0918 20:39:03.408253 24146 ts_manager.cc:186] Registered new tserver with Master: 9ac9d6c4caff4b8dab06322af4fda065 (127.18.222.2:40761) I0918 20:39:03.416031 24397 heartbeater.cc:447] Master 127.18.222.60:46245 was elected leader, sending a full tablet report... I0918 20:39:03.420018 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.016s user 0.010s sys 0.006s I0918 20:39:03.422626 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:03.434397 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:39:03.434650 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/ts-2-root uuid: "18498189d0874c39adf8a1bcd497997a" format_stamp: "Formatted at 2019-09-18 20:39:03 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:03.435024 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1568839089958813-19320/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:03.473093 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:03.473371 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:03.536602 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.3:42283 I0918 20:39:03.539080 19320 webserver.cc:374] Webserver started at http://127.18.222.3:34507/ using document root and password file I0918 20:39:03.570549 24468 heartbeater.cc:322] Connected to a master server at 127.18.222.60:46245 I0918 20:39:03.571035 24468 heartbeater.cc:406] Registering TS with master... I0918 20:39:03.571794 24468 heartbeater.cc:455] Master 127.18.222.60:46245 requested a full tablet report, sending... I0918 20:39:03.571983 24470 heartbeater.cc:322] Connected to a master server at 127.18.222.62:43565 I0918 20:39:03.572382 24470 heartbeater.cc:406] Registering TS with master... I0918 20:39:03.572970 24470 heartbeater.cc:455] Master 127.18.222.62:43565 requested a full tablet report, sending... I0918 20:39:03.573793 24204 ts_manager.cc:186] Registered new tserver with Master: 18498189d0874c39adf8a1bcd497997a (127.18.222.3:42283) I0918 20:39:03.575588 24089 ts_manager.cc:186] Registered new tserver with Master: 18498189d0874c39adf8a1bcd497997a (127.18.222.3:42283) I0918 20:39:03.576325 24204 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49450 I0918 20:39:03.583829 24469 heartbeater.cc:322] Connected to a master server at 127.18.222.61:39201 I0918 20:39:03.584125 24469 heartbeater.cc:406] Registering TS with master... I0918 20:39:03.584483 24469 heartbeater.cc:455] Master 127.18.222.61:39201 requested a full tablet report, sending... I0918 20:39:03.586958 24145 ts_manager.cc:186] Registered new tserver with Master: 18498189d0874c39adf8a1bcd497997a (127.18.222.3:42283) I0918 20:39:03.587833 19320 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.028239529s I0918 20:39:03.593341 24468 heartbeater.cc:447] Master 127.18.222.60:46245 was elected leader, sending a full tablet report... I0918 20:39:03.650146 24202 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49460: name: "ksck-test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\n\000\000\000\004\001\024\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:39:03.651912 24202 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:39:03.765094 24364 tablet_service.cc:1022] Processing CreateTablet for tablet 0602be5295b54b65ae9325757f5443af (table=ksck-test-table [id=a4f242a4ca084f94b9e61f19e3fac22a]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:03.766328 24364 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 0602be5295b54b65ae9325757f5443af. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:03.769479 24363 tablet_service.cc:1022] Processing CreateTablet for tablet f8c2642ea0304e688341b21aa88ecb49 (table=ksck-test-table [id=a4f242a4ca084f94b9e61f19e3fac22a]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:03.770604 24363 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f8c2642ea0304e688341b21aa88ecb49. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:03.783804 24362 tablet_service.cc:1022] Processing CreateTablet for tablet 384e39e1a8be4cabbdc4819e81ab4ea2 (table=ksck-test-table [id=a4f242a4ca084f94b9e61f19e3fac22a]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:03.784938 24362 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 384e39e1a8be4cabbdc4819e81ab4ea2. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:03.828855 24435 tablet_service.cc:1022] Processing CreateTablet for tablet 0602be5295b54b65ae9325757f5443af (table=ksck-test-table [id=a4f242a4ca084f94b9e61f19e3fac22a]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:03.830092 24435 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 0602be5295b54b65ae9325757f5443af. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:03.841012 24434 tablet_service.cc:1022] Processing CreateTablet for tablet f8c2642ea0304e688341b21aa88ecb49 (table=ksck-test-table [id=a4f242a4ca084f94b9e61f19e3fac22a]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:03.842201 24434 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f8c2642ea0304e688341b21aa88ecb49. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:03.853366 24433 tablet_service.cc:1022] Processing CreateTablet for tablet 384e39e1a8be4cabbdc4819e81ab4ea2 (table=ksck-test-table [id=a4f242a4ca084f94b9e61f19e3fac22a]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:03.854605 24433 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 384e39e1a8be4cabbdc4819e81ab4ea2. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:03.859027 24292 tablet_service.cc:1022] Processing CreateTablet for tablet 0602be5295b54b65ae9325757f5443af (table=ksck-test-table [id=a4f242a4ca084f94b9e61f19e3fac22a]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:03.860496 24292 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 0602be5295b54b65ae9325757f5443af. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:03.872594 24489 tablet_bootstrap.cc:439] T 0602be5295b54b65ae9325757f5443af P 9ac9d6c4caff4b8dab06322af4fda065: Bootstrap starting. I0918 20:39:03.875808 24490 tablet_bootstrap.cc:439] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a: Bootstrap starting. I0918 20:39:03.878907 24490 tablet_bootstrap.cc:590] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a: No blocks or log segments found. Creating new log. I0918 20:39:03.882354 24290 tablet_service.cc:1022] Processing CreateTablet for tablet f8c2642ea0304e688341b21aa88ecb49 (table=ksck-test-table [id=a4f242a4ca084f94b9e61f19e3fac22a]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:03.889427 24290 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f8c2642ea0304e688341b21aa88ecb49. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:03.887460 24291 tablet_service.cc:1022] Processing CreateTablet for tablet 384e39e1a8be4cabbdc4819e81ab4ea2 (table=ksck-test-table [id=a4f242a4ca084f94b9e61f19e3fac22a]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:03.884263 24489 tablet_bootstrap.cc:590] T 0602be5295b54b65ae9325757f5443af P 9ac9d6c4caff4b8dab06322af4fda065: No blocks or log segments found. Creating new log. I0918 20:39:03.890241 24291 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 384e39e1a8be4cabbdc4819e81ab4ea2. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:03.913187 24491 tablet_bootstrap.cc:439] T 0602be5295b54b65ae9325757f5443af P 18903fadbbe345ee98fd47291e7835f4: Bootstrap starting. I0918 20:39:03.922590 24489 tablet_bootstrap.cc:439] T 0602be5295b54b65ae9325757f5443af P 9ac9d6c4caff4b8dab06322af4fda065: No bootstrap required, opened a new log I0918 20:39:03.923076 24489 ts_tablet_manager.cc:1085] T 0602be5295b54b65ae9325757f5443af P 9ac9d6c4caff4b8dab06322af4fda065: Time spent bootstrapping tablet: real 0.051s user 0.007s sys 0.004s I0918 20:39:03.927114 24489 raft_consensus.cc:340] T 0602be5295b54b65ae9325757f5443af P 9ac9d6c4caff4b8dab06322af4fda065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } I0918 20:39:03.928189 24489 raft_consensus.cc:366] T 0602be5295b54b65ae9325757f5443af P 9ac9d6c4caff4b8dab06322af4fda065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:03.928483 24489 raft_consensus.cc:705] T 0602be5295b54b65ae9325757f5443af P 9ac9d6c4caff4b8dab06322af4fda065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac9d6c4caff4b8dab06322af4fda065, State: Initialized, Role: FOLLOWER I0918 20:39:03.929333 24489 consensus_queue.cc:229] T 0602be5295b54b65ae9325757f5443af P 9ac9d6c4caff4b8dab06322af4fda065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } I0918 20:39:03.934211 24491 tablet_bootstrap.cc:590] T 0602be5295b54b65ae9325757f5443af P 18903fadbbe345ee98fd47291e7835f4: No blocks or log segments found. Creating new log. I0918 20:39:03.950229 24490 tablet_bootstrap.cc:439] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a: No bootstrap required, opened a new log I0918 20:39:03.950907 24490 ts_tablet_manager.cc:1085] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a: Time spent bootstrapping tablet: real 0.079s user 0.008s sys 0.000s I0918 20:39:03.952901 24489 ts_tablet_manager.cc:1113] T 0602be5295b54b65ae9325757f5443af P 9ac9d6c4caff4b8dab06322af4fda065: Time spent starting tablet: real 0.030s user 0.009s sys 0.010s I0918 20:39:03.953862 24489 tablet_bootstrap.cc:439] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065: Bootstrap starting. I0918 20:39:03.957473 24489 tablet_bootstrap.cc:590] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065: No blocks or log segments found. Creating new log. I0918 20:39:03.958242 24491 tablet_bootstrap.cc:439] T 0602be5295b54b65ae9325757f5443af P 18903fadbbe345ee98fd47291e7835f4: No bootstrap required, opened a new log I0918 20:39:03.958820 24491 ts_tablet_manager.cc:1085] T 0602be5295b54b65ae9325757f5443af P 18903fadbbe345ee98fd47291e7835f4: Time spent bootstrapping tablet: real 0.049s user 0.009s sys 0.015s I0918 20:39:03.958595 24490 raft_consensus.cc:340] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } I0918 20:39:03.959791 24490 raft_consensus.cc:366] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:03.960242 24490 raft_consensus.cc:705] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18498189d0874c39adf8a1bcd497997a, State: Initialized, Role: FOLLOWER I0918 20:39:03.960914 24490 consensus_queue.cc:229] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } I0918 20:39:03.970497 24491 raft_consensus.cc:340] T 0602be5295b54b65ae9325757f5443af P 18903fadbbe345ee98fd47291e7835f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } I0918 20:39:03.979532 24490 ts_tablet_manager.cc:1113] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a: Time spent starting tablet: real 0.028s user 0.008s sys 0.004s I0918 20:39:03.981462 24491 raft_consensus.cc:366] T 0602be5295b54b65ae9325757f5443af P 18903fadbbe345ee98fd47291e7835f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:03.981812 24491 raft_consensus.cc:705] T 0602be5295b54b65ae9325757f5443af P 18903fadbbe345ee98fd47291e7835f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18903fadbbe345ee98fd47291e7835f4, State: Initialized, Role: FOLLOWER I0918 20:39:03.982237 24490 tablet_bootstrap.cc:439] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a: Bootstrap starting. I0918 20:39:03.982741 24491 consensus_queue.cc:229] T 0602be5295b54b65ae9325757f5443af P 18903fadbbe345ee98fd47291e7835f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } I0918 20:39:03.980825 24489 tablet_bootstrap.cc:439] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065: No bootstrap required, opened a new log I0918 20:39:03.984144 24489 ts_tablet_manager.cc:1085] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.007s I0918 20:39:03.986852 24490 tablet_bootstrap.cc:590] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a: No blocks or log segments found. Creating new log. I0918 20:39:03.986107 24489 raft_consensus.cc:340] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } I0918 20:39:04.003266 24489 raft_consensus.cc:366] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:04.003593 24489 raft_consensus.cc:705] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac9d6c4caff4b8dab06322af4fda065, State: Initialized, Role: FOLLOWER I0918 20:39:04.008925 24247 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7: acquiring CA information for follower catalog manager: success I0918 20:39:04.012156 24249 catalog_manager.cc:1153] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2: acquiring CA information for follower catalog manager: success I0918 20:39:04.020428 24249 catalog_manager.cc:1181] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:04.021489 24247 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:04.007344 24489 consensus_queue.cc:229] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } I0918 20:39:04.030930 24490 tablet_bootstrap.cc:439] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a: No bootstrap required, opened a new log I0918 20:39:04.031473 24490 ts_tablet_manager.cc:1085] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a: Time spent bootstrapping tablet: real 0.049s user 0.018s sys 0.014s I0918 20:39:04.033203 24491 ts_tablet_manager.cc:1113] T 0602be5295b54b65ae9325757f5443af P 18903fadbbe345ee98fd47291e7835f4: Time spent starting tablet: real 0.074s user 0.014s sys 0.000s I0918 20:39:04.040155 24491 tablet_bootstrap.cc:439] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18903fadbbe345ee98fd47291e7835f4: Bootstrap starting. I0918 20:39:04.041677 24489 ts_tablet_manager.cc:1113] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065: Time spent starting tablet: real 0.057s user 0.001s sys 0.005s I0918 20:39:04.042785 24489 tablet_bootstrap.cc:439] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 9ac9d6c4caff4b8dab06322af4fda065: Bootstrap starting. I0918 20:39:04.043845 24490 raft_consensus.cc:340] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } I0918 20:39:04.044680 24490 raft_consensus.cc:366] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:04.045639 24489 tablet_bootstrap.cc:590] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 9ac9d6c4caff4b8dab06322af4fda065: No blocks or log segments found. Creating new log. I0918 20:39:04.045769 24491 tablet_bootstrap.cc:590] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18903fadbbe345ee98fd47291e7835f4: No blocks or log segments found. Creating new log. I0918 20:39:04.046000 24490 raft_consensus.cc:705] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18498189d0874c39adf8a1bcd497997a, State: Initialized, Role: FOLLOWER I0918 20:39:04.046905 24490 consensus_queue.cc:229] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } I0918 20:39:04.052687 24490 ts_tablet_manager.cc:1113] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a: Time spent starting tablet: real 0.021s user 0.003s sys 0.004s I0918 20:39:04.053604 24490 tablet_bootstrap.cc:439] T f8c2642ea0304e688341b21aa88ecb49 P 18498189d0874c39adf8a1bcd497997a: Bootstrap starting. I0918 20:39:04.056336 24497 raft_consensus.cc:472] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:04.056823 24497 raft_consensus.cc:494] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } I0918 20:39:04.057301 24490 tablet_bootstrap.cc:590] T f8c2642ea0304e688341b21aa88ecb49 P 18498189d0874c39adf8a1bcd497997a: No blocks or log segments found. Creating new log. I0918 20:39:04.058710 24497 leader_election.cc:296] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ac9d6c4caff4b8dab06322af4fda065 (127.18.222.2:40761), 18903fadbbe345ee98fd47291e7835f4 (127.18.222.1:33561) I0918 20:39:04.061539 24498 raft_consensus.cc:472] T 0602be5295b54b65ae9325757f5443af P 18903fadbbe345ee98fd47291e7835f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:04.062377 24498 raft_consensus.cc:494] T 0602be5295b54b65ae9325757f5443af P 18903fadbbe345ee98fd47291e7835f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } I0918 20:39:04.066227 24498 leader_election.cc:296] T 0602be5295b54b65ae9325757f5443af P 18903fadbbe345ee98fd47291e7835f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ac9d6c4caff4b8dab06322af4fda065 (127.18.222.2:40761), 18498189d0874c39adf8a1bcd497997a (127.18.222.3:42283) I0918 20:39:04.086844 24489 tablet_bootstrap.cc:439] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 9ac9d6c4caff4b8dab06322af4fda065: No bootstrap required, opened a new log I0918 20:39:04.087374 24489 ts_tablet_manager.cc:1085] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 9ac9d6c4caff4b8dab06322af4fda065: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.001s I0918 20:39:04.090021 24491 tablet_bootstrap.cc:439] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18903fadbbe345ee98fd47291e7835f4: No bootstrap required, opened a new log I0918 20:39:04.090880 24491 ts_tablet_manager.cc:1085] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18903fadbbe345ee98fd47291e7835f4: Time spent bootstrapping tablet: real 0.051s user 0.007s sys 0.000s I0918 20:39:04.093408 24491 raft_consensus.cc:340] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18903fadbbe345ee98fd47291e7835f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } I0918 20:39:04.094558 24491 raft_consensus.cc:366] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18903fadbbe345ee98fd47291e7835f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:04.102107 24491 raft_consensus.cc:705] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18903fadbbe345ee98fd47291e7835f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18903fadbbe345ee98fd47291e7835f4, State: Initialized, Role: FOLLOWER I0918 20:39:04.113461 24374 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0602be5295b54b65ae9325757f5443af" candidate_uuid: "18498189d0874c39adf8a1bcd497997a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac9d6c4caff4b8dab06322af4fda065" is_pre_election: true I0918 20:39:04.114114 24374 raft_consensus.cc:2350] T 0602be5295b54b65ae9325757f5443af P 9ac9d6c4caff4b8dab06322af4fda065 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18498189d0874c39adf8a1bcd497997a in term 0. I0918 20:39:04.115499 24407 leader_election.cc:310] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18498189d0874c39adf8a1bcd497997a, 9ac9d6c4caff4b8dab06322af4fda065; no voters: I0918 20:39:04.107594 24491 consensus_queue.cc:229] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18903fadbbe345ee98fd47291e7835f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } I0918 20:39:04.116407 24497 raft_consensus.cc:2644] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:04.117661 24497 raft_consensus.cc:472] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:04.118072 24497 raft_consensus.cc:2890] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:04.125226 24497 raft_consensus.cc:494] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } I0918 20:39:04.130650 24491 ts_tablet_manager.cc:1113] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18903fadbbe345ee98fd47291e7835f4: Time spent starting tablet: real 0.039s user 0.006s sys 0.000s I0918 20:39:04.130822 24490 tablet_bootstrap.cc:439] T f8c2642ea0304e688341b21aa88ecb49 P 18498189d0874c39adf8a1bcd497997a: No bootstrap required, opened a new log I0918 20:39:04.131338 24490 ts_tablet_manager.cc:1085] T f8c2642ea0304e688341b21aa88ecb49 P 18498189d0874c39adf8a1bcd497997a: Time spent bootstrapping tablet: real 0.078s user 0.015s sys 0.029s I0918 20:39:04.132398 24497 leader_election.cc:296] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [CANDIDATE]: Term 1 election: Requested vote from peers 9ac9d6c4caff4b8dab06322af4fda065 (127.18.222.2:40761), 18903fadbbe345ee98fd47291e7835f4 (127.18.222.1:33561) I0918 20:39:04.131683 24491 tablet_bootstrap.cc:439] T f8c2642ea0304e688341b21aa88ecb49 P 18903fadbbe345ee98fd47291e7835f4: Bootstrap starting. I0918 20:39:04.133365 24490 raft_consensus.cc:340] T f8c2642ea0304e688341b21aa88ecb49 P 18498189d0874c39adf8a1bcd497997a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } I0918 20:39:04.134152 24490 raft_consensus.cc:366] T f8c2642ea0304e688341b21aa88ecb49 P 18498189d0874c39adf8a1bcd497997a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:04.134434 24490 raft_consensus.cc:705] T f8c2642ea0304e688341b21aa88ecb49 P 18498189d0874c39adf8a1bcd497997a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18498189d0874c39adf8a1bcd497997a, State: Initialized, Role: FOLLOWER I0918 20:39:04.136184 24489 raft_consensus.cc:340] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 9ac9d6c4caff4b8dab06322af4fda065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } I0918 20:39:04.136781 24491 tablet_bootstrap.cc:590] T f8c2642ea0304e688341b21aa88ecb49 P 18903fadbbe345ee98fd47291e7835f4: No blocks or log segments found. Creating new log. I0918 20:39:04.137066 24490 consensus_queue.cc:229] T f8c2642ea0304e688341b21aa88ecb49 P 18498189d0874c39adf8a1bcd497997a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } I0918 20:39:04.138396 24490 ts_tablet_manager.cc:1113] T f8c2642ea0304e688341b21aa88ecb49 P 18498189d0874c39adf8a1bcd497997a: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s I0918 20:39:04.136795 24489 raft_consensus.cc:366] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 9ac9d6c4caff4b8dab06322af4fda065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:04.140401 24489 raft_consensus.cc:705] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 9ac9d6c4caff4b8dab06322af4fda065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac9d6c4caff4b8dab06322af4fda065, State: Initialized, Role: FOLLOWER I0918 20:39:04.141352 24489 consensus_queue.cc:229] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 9ac9d6c4caff4b8dab06322af4fda065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } I0918 20:39:04.143545 24489 ts_tablet_manager.cc:1113] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 9ac9d6c4caff4b8dab06322af4fda065: Time spent starting tablet: real 0.042s user 0.003s sys 0.003s I0918 20:39:04.140385 24374 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0602be5295b54b65ae9325757f5443af" candidate_uuid: "18498189d0874c39adf8a1bcd497997a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac9d6c4caff4b8dab06322af4fda065" I0918 20:39:04.147521 24491 tablet_bootstrap.cc:439] T f8c2642ea0304e688341b21aa88ecb49 P 18903fadbbe345ee98fd47291e7835f4: No bootstrap required, opened a new log I0918 20:39:04.148152 24491 ts_tablet_manager.cc:1085] T f8c2642ea0304e688341b21aa88ecb49 P 18903fadbbe345ee98fd47291e7835f4: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.000s I0918 20:39:04.147662 24374 raft_consensus.cc:2890] T 0602be5295b54b65ae9325757f5443af P 9ac9d6c4caff4b8dab06322af4fda065 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:04.169971 24491 raft_consensus.cc:340] T f8c2642ea0304e688341b21aa88ecb49 P 18903fadbbe345ee98fd47291e7835f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } I0918 20:39:04.173233 24491 raft_consensus.cc:366] T f8c2642ea0304e688341b21aa88ecb49 P 18903fadbbe345ee98fd47291e7835f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:04.173461 24491 raft_consensus.cc:705] T f8c2642ea0304e688341b21aa88ecb49 P 18903fadbbe345ee98fd47291e7835f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18903fadbbe345ee98fd47291e7835f4, State: Initialized, Role: FOLLOWER I0918 20:39:04.174224 24491 consensus_queue.cc:229] T f8c2642ea0304e688341b21aa88ecb49 P 18903fadbbe345ee98fd47291e7835f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } I0918 20:39:04.176550 24497 raft_consensus.cc:472] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:04.177031 24497 raft_consensus.cc:494] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } I0918 20:39:04.177451 24491 ts_tablet_manager.cc:1113] T f8c2642ea0304e688341b21aa88ecb49 P 18903fadbbe345ee98fd47291e7835f4: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s I0918 20:39:04.178551 24497 leader_election.cc:296] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18903fadbbe345ee98fd47291e7835f4 (127.18.222.1:33561), 9ac9d6c4caff4b8dab06322af4fda065 (127.18.222.2:40761) I0918 20:39:04.179127 24494 raft_consensus.cc:472] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:04.179626 24494 raft_consensus.cc:494] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } I0918 20:39:04.181548 24494 leader_election.cc:296] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18903fadbbe345ee98fd47291e7835f4 (127.18.222.1:33561), 18498189d0874c39adf8a1bcd497997a (127.18.222.3:42283) I0918 20:39:04.186406 24445 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0602be5295b54b65ae9325757f5443af" candidate_uuid: "18903fadbbe345ee98fd47291e7835f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18498189d0874c39adf8a1bcd497997a" is_pre_election: true I0918 20:39:04.187289 24445 raft_consensus.cc:2280] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 18903fadbbe345ee98fd47291e7835f4 in current term 1: Already voted for candidate 18498189d0874c39adf8a1bcd497997a in this term. I0918 20:39:04.216137 24372 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0602be5295b54b65ae9325757f5443af" candidate_uuid: "18903fadbbe345ee98fd47291e7835f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac9d6c4caff4b8dab06322af4fda065" is_pre_election: true I0918 20:39:04.216040 24373 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "384e39e1a8be4cabbdc4819e81ab4ea2" candidate_uuid: "18498189d0874c39adf8a1bcd497997a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac9d6c4caff4b8dab06322af4fda065" is_pre_election: true I0918 20:39:04.216996 24373 raft_consensus.cc:2350] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 9ac9d6c4caff4b8dab06322af4fda065 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18498189d0874c39adf8a1bcd497997a in term 0. I0918 20:39:04.217218 24303 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0602be5295b54b65ae9325757f5443af" candidate_uuid: "18498189d0874c39adf8a1bcd497997a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18903fadbbe345ee98fd47291e7835f4" is_pre_election: true I0918 20:39:04.217955 24303 raft_consensus.cc:2350] T 0602be5295b54b65ae9325757f5443af P 18903fadbbe345ee98fd47291e7835f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18498189d0874c39adf8a1bcd497997a in term 0. I0918 20:39:04.224124 24303 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0602be5295b54b65ae9325757f5443af" candidate_uuid: "18498189d0874c39adf8a1bcd497997a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18903fadbbe345ee98fd47291e7835f4" I0918 20:39:04.224751 24303 raft_consensus.cc:2890] T 0602be5295b54b65ae9325757f5443af P 18903fadbbe345ee98fd47291e7835f4 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:04.224601 24302 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "384e39e1a8be4cabbdc4819e81ab4ea2" candidate_uuid: "18498189d0874c39adf8a1bcd497997a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18903fadbbe345ee98fd47291e7835f4" is_pre_election: true I0918 20:39:04.225446 24302 raft_consensus.cc:2350] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18903fadbbe345ee98fd47291e7835f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18498189d0874c39adf8a1bcd497997a in term 0. I0918 20:39:04.211001 24374 raft_consensus.cc:2350] T 0602be5295b54b65ae9325757f5443af P 9ac9d6c4caff4b8dab06322af4fda065 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18498189d0874c39adf8a1bcd497997a in term 1. I0918 20:39:04.224746 24407 leader_election.cc:310] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18498189d0874c39adf8a1bcd497997a, 9ac9d6c4caff4b8dab06322af4fda065; no voters: I0918 20:39:04.228380 24372 raft_consensus.cc:2325] T 0602be5295b54b65ae9325757f5443af P 9ac9d6c4caff4b8dab06322af4fda065 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 18903fadbbe345ee98fd47291e7835f4 for term 1 because replica is already servicing an update from a current leader or another vote. I0918 20:39:04.228780 24497 raft_consensus.cc:2644] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:04.229187 24497 raft_consensus.cc:472] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:04.229506 24497 raft_consensus.cc:2890] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:04.230129 24253 leader_election.cc:310] T 0602be5295b54b65ae9325757f5443af P 18903fadbbe345ee98fd47291e7835f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18903fadbbe345ee98fd47291e7835f4; no voters: 18498189d0874c39adf8a1bcd497997a, 9ac9d6c4caff4b8dab06322af4fda065 I0918 20:39:04.236272 24303 raft_consensus.cc:2350] T 0602be5295b54b65ae9325757f5443af P 18903fadbbe345ee98fd47291e7835f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18498189d0874c39adf8a1bcd497997a in term 1. I0918 20:39:04.236959 24498 raft_consensus.cc:2592] T 0602be5295b54b65ae9325757f5443af P 18903fadbbe345ee98fd47291e7835f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0918 20:39:04.237200 24497 raft_consensus.cc:494] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } I0918 20:39:04.239598 24497 leader_election.cc:296] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a [CANDIDATE]: Term 1 election: Requested vote from peers 18903fadbbe345ee98fd47291e7835f4 (127.18.222.1:33561), 9ac9d6c4caff4b8dab06322af4fda065 (127.18.222.2:40761) I0918 20:39:04.243636 24303 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f8c2642ea0304e688341b21aa88ecb49" candidate_uuid: "9ac9d6c4caff4b8dab06322af4fda065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18903fadbbe345ee98fd47291e7835f4" is_pre_election: true I0918 20:39:04.244698 24303 raft_consensus.cc:2350] T f8c2642ea0304e688341b21aa88ecb49 P 18903fadbbe345ee98fd47291e7835f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ac9d6c4caff4b8dab06322af4fda065 in term 0. I0918 20:39:04.245882 24336 leader_election.cc:310] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18903fadbbe345ee98fd47291e7835f4, 9ac9d6c4caff4b8dab06322af4fda065; no voters: I0918 20:39:04.246721 24494 raft_consensus.cc:2644] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:04.247030 24494 raft_consensus.cc:472] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:04.247059 24445 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f8c2642ea0304e688341b21aa88ecb49" candidate_uuid: "9ac9d6c4caff4b8dab06322af4fda065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18498189d0874c39adf8a1bcd497997a" is_pre_election: true I0918 20:39:04.247830 24445 raft_consensus.cc:2350] T f8c2642ea0304e688341b21aa88ecb49 P 18498189d0874c39adf8a1bcd497997a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ac9d6c4caff4b8dab06322af4fda065 in term 0. I0918 20:39:04.247370 24494 raft_consensus.cc:2890] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:04.248320 24407 leader_election.cc:310] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18498189d0874c39adf8a1bcd497997a, 9ac9d6c4caff4b8dab06322af4fda065; no voters: I0918 20:39:04.249912 24497 raft_consensus.cc:2644] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:04.251919 24374 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "384e39e1a8be4cabbdc4819e81ab4ea2" candidate_uuid: "18498189d0874c39adf8a1bcd497997a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac9d6c4caff4b8dab06322af4fda065" I0918 20:39:04.252552 24374 raft_consensus.cc:2890] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 9ac9d6c4caff4b8dab06322af4fda065 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:04.255323 24494 raft_consensus.cc:494] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } I0918 20:39:04.257033 24494 leader_election.cc:296] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 [CANDIDATE]: Term 1 election: Requested vote from peers 18903fadbbe345ee98fd47291e7835f4 (127.18.222.1:33561), 18498189d0874c39adf8a1bcd497997a (127.18.222.3:42283) I0918 20:39:04.257954 24497 raft_consensus.cc:667] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [term 1 LEADER]: Becoming Leader. State: Replica: 18498189d0874c39adf8a1bcd497997a, State: Running, Role: LEADER I0918 20:39:04.258502 24445 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f8c2642ea0304e688341b21aa88ecb49" candidate_uuid: "9ac9d6c4caff4b8dab06322af4fda065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18498189d0874c39adf8a1bcd497997a" I0918 20:39:04.259135 24445 raft_consensus.cc:2890] T f8c2642ea0304e688341b21aa88ecb49 P 18498189d0874c39adf8a1bcd497997a [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:04.260630 24374 raft_consensus.cc:2350] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 9ac9d6c4caff4b8dab06322af4fda065 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18498189d0874c39adf8a1bcd497997a in term 1. I0918 20:39:04.261234 24303 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f8c2642ea0304e688341b21aa88ecb49" candidate_uuid: "9ac9d6c4caff4b8dab06322af4fda065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18903fadbbe345ee98fd47291e7835f4" I0918 20:39:04.261919 24303 raft_consensus.cc:2890] T f8c2642ea0304e688341b21aa88ecb49 P 18903fadbbe345ee98fd47291e7835f4 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:04.261739 24302 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "384e39e1a8be4cabbdc4819e81ab4ea2" candidate_uuid: "18498189d0874c39adf8a1bcd497997a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18903fadbbe345ee98fd47291e7835f4" I0918 20:39:04.262408 24302 raft_consensus.cc:2890] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18903fadbbe345ee98fd47291e7835f4 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:04.264051 24445 raft_consensus.cc:2350] T f8c2642ea0304e688341b21aa88ecb49 P 18498189d0874c39adf8a1bcd497997a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ac9d6c4caff4b8dab06322af4fda065 in term 1. I0918 20:39:04.265583 24337 leader_election.cc:310] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18498189d0874c39adf8a1bcd497997a, 9ac9d6c4caff4b8dab06322af4fda065; no voters: I0918 20:39:04.266566 24494 raft_consensus.cc:2644] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:04.259173 24497 consensus_queue.cc:206] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } I0918 20:39:04.270938 24302 raft_consensus.cc:2350] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18903fadbbe345ee98fd47291e7835f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18498189d0874c39adf8a1bcd497997a in term 1. I0918 20:39:04.275080 24407 leader_election.cc:310] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18498189d0874c39adf8a1bcd497997a, 9ac9d6c4caff4b8dab06322af4fda065; no voters: I0918 20:39:04.275970 24511 raft_consensus.cc:2644] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:04.279914 24494 raft_consensus.cc:667] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 [term 1 LEADER]: Becoming Leader. State: Replica: 9ac9d6c4caff4b8dab06322af4fda065, State: Running, Role: LEADER I0918 20:39:04.281590 24303 raft_consensus.cc:2350] T f8c2642ea0304e688341b21aa88ecb49 P 18903fadbbe345ee98fd47291e7835f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ac9d6c4caff4b8dab06322af4fda065 in term 1. I0918 20:39:04.281026 24494 consensus_queue.cc:206] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } I0918 20:39:04.283605 24511 raft_consensus.cc:667] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a [term 1 LEADER]: Becoming Leader. State: Replica: 18498189d0874c39adf8a1bcd497997a, State: Running, Role: LEADER I0918 20:39:04.289573 24511 consensus_queue.cc:206] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } } peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } } I0918 20:39:04.295543 24201 catalog_manager.cc:4131] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a reported cstate change: term changed from 0 to 1, leader changed from to 18498189d0874c39adf8a1bcd497997a (127.18.222.3). New cstate: current_term: 1 leader_uuid: "18498189d0874c39adf8a1bcd497997a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } health_report { overall_health: UNKNOWN } } } I0918 20:39:04.299538 24203 catalog_manager.cc:4131] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 reported cstate change: term changed from 0 to 1, leader changed from to 9ac9d6c4caff4b8dab06322af4fda065 (127.18.222.2). New cstate: current_term: 1 leader_uuid: "9ac9d6c4caff4b8dab06322af4fda065" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } health_report { overall_health: UNKNOWN } } } I0918 20:39:04.344126 24201 catalog_manager.cc:4131] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a reported cstate change: term changed from 0 to 1, leader changed from to 18498189d0874c39adf8a1bcd497997a (127.18.222.3). New cstate: current_term: 1 leader_uuid: "18498189d0874c39adf8a1bcd497997a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 } health_report { overall_health: HEALTHY } } } I0918 20:39:04.675149 24494 consensus_queue.cc:984] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18498189d0874c39adf8a1bcd497997a" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:04.680814 24512 consensus_queue.cc:984] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s I0918 20:39:04.746016 24511 consensus_queue.cc:984] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:04.782402 24511 consensus_queue.cc:984] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:04.895404 24511 consensus_queue.cc:984] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac9d6c4caff4b8dab06322af4fda065" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:04.940927 24554 consensus_queue.cc:984] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18903fadbbe345ee98fd47291e7835f4" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 33561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s I0918 20:39:04.994421 19320 master.cc:247] Master@127.18.222.60:46245 shutting down... I0918 20:39:04.995344 24186 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:05.004082 24188 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:05.013025 24187 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:05.014081 24188 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:05.014891 24186 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:05.028894 24187 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:05.029181 24186 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:05.036792 24188 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:05.044672 24187 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:05.045874 24186 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:05.050618 24188 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:05.059298 24187 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:05.060256 24186 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:05.065058 24188 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:05.067538 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:05.068504 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:05.068789 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 2fe1f99a02a849dca2a9e9aa4ed858dc: stopping tablet replica I0918 20:39:05.072697 24187 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:05.074774 24186 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:05.077895 24188 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:05.100632 19320 master.cc:258] Master@127.18.222.60:46245 shutdown complete. I0918 20:39:06.340607 24572 raft_consensus.cc:472] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2fe1f99a02a849dca2a9e9aa4ed858dc) I0918 20:39:06.341159 24572 raft_consensus.cc:494] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } I0918 20:39:06.342857 24572 leader_election.cc:296] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b69dbcabcdda4e728f542c193ca7b1b2 (127.18.222.61:39201), 2fe1f99a02a849dca2a9e9aa4ed858dc (127.18.222.60:46245) I0918 20:39:06.343667 24156 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "17f33eef4f4c4000a2dc645ff32512d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" is_pre_election: true I0918 20:39:06.344667 24156 raft_consensus.cc:2311] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17f33eef4f4c4000a2dc645ff32512d7 for term 2 because replica is either leader or believes a valid leader to be alive. W0918 20:39:06.345911 24071 leader_election.cc:341] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2fe1f99a02a849dca2a9e9aa4ed858dc (127.18.222.60:46245): Network error: Client connection negotiation failed: client connection to 127.18.222.60:46245: connect: Connection refused (error 111) I0918 20:39:06.346719 24070 leader_election.cc:310] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17f33eef4f4c4000a2dc645ff32512d7; no voters: 2fe1f99a02a849dca2a9e9aa4ed858dc, b69dbcabcdda4e728f542c193ca7b1b2 I0918 20:39:06.347487 24572 raft_consensus.cc:2592] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0918 20:39:06.468705 24574 raft_consensus.cc:472] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2fe1f99a02a849dca2a9e9aa4ed858dc) I0918 20:39:06.469126 24574 raft_consensus.cc:494] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } I0918 20:39:06.470829 24574 leader_election.cc:296] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 17f33eef4f4c4000a2dc645ff32512d7 (127.18.222.62:43565), 2fe1f99a02a849dca2a9e9aa4ed858dc (127.18.222.60:46245) I0918 20:39:06.471508 24098 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "17f33eef4f4c4000a2dc645ff32512d7" is_pre_election: true I0918 20:39:06.472402 24098 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b69dbcabcdda4e728f542c193ca7b1b2 in term 1. W0918 20:39:06.473170 24128 leader_election.cc:341] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2fe1f99a02a849dca2a9e9aa4ed858dc (127.18.222.60:46245): Network error: Client connection negotiation failed: client connection to 127.18.222.60:46245: connect: Connection refused (error 111) I0918 20:39:06.473897 24128 leader_election.cc:310] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17f33eef4f4c4000a2dc645ff32512d7, b69dbcabcdda4e728f542c193ca7b1b2; no voters: 2fe1f99a02a849dca2a9e9aa4ed858dc I0918 20:39:06.475091 24574 raft_consensus.cc:2644] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0918 20:39:06.475433 24574 raft_consensus.cc:472] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2fe1f99a02a849dca2a9e9aa4ed858dc) I0918 20:39:06.475891 24574 raft_consensus.cc:2890] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [term 1 FOLLOWER]: Advancing to term 2 I0918 20:39:06.481504 24574 raft_consensus.cc:494] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } I0918 20:39:06.483994 24098 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "17f33eef4f4c4000a2dc645ff32512d7" I0918 20:39:06.484617 24098 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [term 1 FOLLOWER]: Advancing to term 2 W0918 20:39:06.486959 24128 leader_election.cc:341] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2fe1f99a02a849dca2a9e9aa4ed858dc (127.18.222.60:46245): Network error: Client connection negotiation failed: client connection to 127.18.222.60:46245: connect: Connection refused (error 111) I0918 20:39:06.486953 24574 leader_election.cc:296] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [CANDIDATE]: Term 2 election: Requested vote from peers 17f33eef4f4c4000a2dc645ff32512d7 (127.18.222.62:43565), 2fe1f99a02a849dca2a9e9aa4ed858dc (127.18.222.60:46245) I0918 20:39:06.490013 24098 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b69dbcabcdda4e728f542c193ca7b1b2 in term 2. I0918 20:39:06.490944 24128 leader_election.cc:310] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17f33eef4f4c4000a2dc645ff32512d7, b69dbcabcdda4e728f542c193ca7b1b2; no voters: 2fe1f99a02a849dca2a9e9aa4ed858dc I0918 20:39:06.492738 24574 raft_consensus.cc:2644] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [term 2 FOLLOWER]: Leader election won for term 2 I0918 20:39:06.495005 24574 raft_consensus.cc:667] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [term 2 LEADER]: Becoming Leader. State: Replica: b69dbcabcdda4e728f542c193ca7b1b2, State: Running, Role: LEADER I0918 20:39:06.495810 24574 consensus_queue.cc:206] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } I0918 20:39:06.501700 24576 sys_catalog.cc:340] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b69dbcabcdda4e728f542c193ca7b1b2. Latest consensus state: current_term: 2 leader_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f33eef4f4c4000a2dc645ff32512d7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43565 } } peers { permanent_uuid: "b69dbcabcdda4e728f542c193ca7b1b2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39201 } } peers { permanent_uuid: "2fe1f99a02a849dca2a9e9aa4ed858dc" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 46245 } } } I0918 20:39:06.507611 24576 sys_catalog.cc:343] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [sys.catalog]: This master's current role is: LEADER I0918 20:39:06.513016 24578 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:39:06.517441 24578 catalog_manager.cc:387] Loaded metadata for table ksck-test-table [id=a4f242a4ca084f94b9e61f19e3fac22a] I0918 20:39:06.524116 24578 catalog_manager.cc:441] Loaded metadata for tablet 0602be5295b54b65ae9325757f5443af (table ksck-test-table [id=a4f242a4ca084f94b9e61f19e3fac22a]) I0918 20:39:06.525110 24578 catalog_manager.cc:441] Loaded metadata for tablet 384e39e1a8be4cabbdc4819e81ab4ea2 (table ksck-test-table [id=a4f242a4ca084f94b9e61f19e3fac22a]) I0918 20:39:06.526021 24578 catalog_manager.cc:441] Loaded metadata for tablet f8c2642ea0304e688341b21aa88ecb49 (table ksck-test-table [id=a4f242a4ca084f94b9e61f19e3fac22a]) I0918 20:39:06.526810 24578 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:39:06.531090 24578 catalog_manager.cc:1106] Loading token signing keys... I0918 20:39:06.534134 24578 catalog_manager.cc:4457] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2: Loaded TSK: 0 I0918 20:39:06.535518 24578 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:39:06.552620 24398 heartbeater.cc:447] Master 127.18.222.61:39201 was elected leader, sending a full tablet report... I0918 20:39:06.553442 24469 heartbeater.cc:447] Master 127.18.222.61:39201 was elected leader, sending a full tablet report... I0918 20:39:06.557121 24327 heartbeater.cc:447] Master 127.18.222.61:39201 was elected leader, sending a full tablet report... I0918 20:39:06.735453 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:06.784431 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:06.784869 19320 tablet_replica.cc:273] T f8c2642ea0304e688341b21aa88ecb49 P 18903fadbbe345ee98fd47291e7835f4: stopping tablet replica I0918 20:39:06.785429 19320 raft_consensus.cc:2147] T f8c2642ea0304e688341b21aa88ecb49 P 18903fadbbe345ee98fd47291e7835f4 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:06.785866 19320 raft_consensus.cc:2174] T f8c2642ea0304e688341b21aa88ecb49 P 18903fadbbe345ee98fd47291e7835f4 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:06.788242 19320 tablet_replica.cc:273] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18903fadbbe345ee98fd47291e7835f4: stopping tablet replica I0918 20:39:06.788568 19320 raft_consensus.cc:2147] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18903fadbbe345ee98fd47291e7835f4 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:06.788877 19320 raft_consensus.cc:2174] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18903fadbbe345ee98fd47291e7835f4 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:06.790839 19320 tablet_replica.cc:273] T 0602be5295b54b65ae9325757f5443af P 18903fadbbe345ee98fd47291e7835f4: stopping tablet replica I0918 20:39:06.791188 19320 raft_consensus.cc:2147] T 0602be5295b54b65ae9325757f5443af P 18903fadbbe345ee98fd47291e7835f4 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:06.791441 19320 raft_consensus.cc:2174] T 0602be5295b54b65ae9325757f5443af P 18903fadbbe345ee98fd47291e7835f4 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:06.815899 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:06.816164 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:06.816566 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:06.829246 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:06.852295 24335 messenger.cc:381] Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer W0918 20:39:06.855304 24407 consensus_peers.cc:458] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a -> Peer 9ac9d6c4caff4b8dab06322af4fda065 (127.18.222.2:40761): Couldn't send request to peer 9ac9d6c4caff4b8dab06322af4fda065. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry. I0918 20:39:06.870165 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:06.870494 19320 tablet_replica.cc:273] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 9ac9d6c4caff4b8dab06322af4fda065: stopping tablet replica I0918 20:39:06.870991 19320 raft_consensus.cc:2147] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 9ac9d6c4caff4b8dab06322af4fda065 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:06.871296 19320 raft_consensus.cc:2174] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 9ac9d6c4caff4b8dab06322af4fda065 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:06.873347 19320 tablet_replica.cc:273] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065: stopping tablet replica I0918 20:39:06.873694 19320 raft_consensus.cc:2147] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:06.874327 19320 raft_consensus.cc:2174] T f8c2642ea0304e688341b21aa88ecb49 P 9ac9d6c4caff4b8dab06322af4fda065 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:06.876646 19320 tablet_replica.cc:273] T 0602be5295b54b65ae9325757f5443af P 9ac9d6c4caff4b8dab06322af4fda065: stopping tablet replica I0918 20:39:06.876991 19320 raft_consensus.cc:2147] T 0602be5295b54b65ae9325757f5443af P 9ac9d6c4caff4b8dab06322af4fda065 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:06.877393 19320 raft_consensus.cc:2174] T 0602be5295b54b65ae9325757f5443af P 9ac9d6c4caff4b8dab06322af4fda065 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:06.905059 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:06.905256 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:06.905654 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:06.922267 19320 tablet_server.cc:155] TabletServer shutting down... W0918 20:39:06.959740 24407 consensus_peers.cc:458] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a -> Peer 18903fadbbe345ee98fd47291e7835f4 (127.18.222.1:33561): Couldn't send request to peer 18903fadbbe345ee98fd47291e7835f4. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.1:33561: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0918 20:39:06.973474 24128 consensus_peers.cc:458] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 -> Peer 2fe1f99a02a849dca2a9e9aa4ed858dc (127.18.222.60:46245): Couldn't send request to peer 2fe1f99a02a849dca2a9e9aa4ed858dc. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.60:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:39:06.992696 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:06.993194 19320 tablet_replica.cc:273] T f8c2642ea0304e688341b21aa88ecb49 P 18498189d0874c39adf8a1bcd497997a: stopping tablet replica I0918 20:39:06.993867 19320 raft_consensus.cc:2147] T f8c2642ea0304e688341b21aa88ecb49 P 18498189d0874c39adf8a1bcd497997a [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:06.994390 19320 raft_consensus.cc:2174] T f8c2642ea0304e688341b21aa88ecb49 P 18498189d0874c39adf8a1bcd497997a [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:06.998399 19320 tablet_replica.cc:273] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a: stopping tablet replica I0918 20:39:06.999060 19320 raft_consensus.cc:2147] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:07.000320 19320 raft_consensus.cc:2174] T 384e39e1a8be4cabbdc4819e81ab4ea2 P 18498189d0874c39adf8a1bcd497997a [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:07.003115 19320 tablet_replica.cc:273] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a: stopping tablet replica I0918 20:39:07.003616 19320 raft_consensus.cc:2147] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:07.004606 19320 raft_consensus.cc:2174] T 0602be5295b54b65ae9325757f5443af P 18498189d0874c39adf8a1bcd497997a [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:07.031193 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:07.031535 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:07.032081 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:07.050359 19320 master.cc:247] Master@127.18.222.62:43565 shutting down... I0918 20:39:07.061012 24070 messenger.cc:381] Service unavailable: service kudu.consensus.ConsensusService not registered on Master W0918 20:39:07.061959 24128 consensus_peers.cc:458] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 -> Peer 17f33eef4f4c4000a2dc645ff32512d7 (127.18.222.62:43565): Couldn't send request to peer 17f33eef4f4c4000a2dc645ff32512d7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry. I0918 20:39:07.081514 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [term 2 FOLLOWER]: Raft consensus shutting down. I0918 20:39:07.082229 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7 [term 2 FOLLOWER]: Raft consensus is shut down! I0918 20:39:07.082687 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 17f33eef4f4c4000a2dc645ff32512d7: stopping tablet replica I0918 20:39:07.105906 19320 master.cc:258] Master@127.18.222.62:43565 shutdown complete. I0918 20:39:07.118592 19320 master.cc:247] Master@127.18.222.61:39201 shutting down... I0918 20:39:07.151830 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [term 2 LEADER]: Raft consensus shutting down. I0918 20:39:07.152781 19320 pending_rounds.cc:64] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2: Trying to abort 1 pending transactions. I0918 20:39:07.152948 19320 pending_rounds.cc:70] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2: Aborting transaction as it isn't in flight: id { term: 2 index: 9 } timestamp: 6425965144054943744 op_type: NO_OP noop_request { } I0918 20:39:07.153410 19320 raft_consensus.cc:2727] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [term 2 LEADER]: NO_OP replication failed: Aborted: Transaction aborted I0918 20:39:07.153780 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2 [term 2 FOLLOWER]: Raft consensus is shut down! I0918 20:39:07.154165 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P b69dbcabcdda4e728f542c193ca7b1b2: stopping tablet replica I0918 20:39:07.179832 19320 master.cc:258] Master@127.18.222.61:39201 shutdown complete. [ OK ] RemoteKsckTest.TestLeaderMasterDown (4719 ms) [ RUN ] RemoteKsckTest.TestClusterWithLocation I0918 20:39:07.232296 19320 test_util.cc:212] Using random seed: -736846846 I0918 20:39:07.233193 19320 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.18.222.62:37057,127.18.222.61:42813,127.18.222.60:45381 I0918 20:39:07.250900 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 151092us I0918 20:39:07.251497 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:07.252056 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:07.252312 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:07.252501 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:07.253799 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-0-root/instance: uuid: "c099560cc34545d88ee0f5408bf0078e" format_stamp: "Formatted at 2019-09-18 20:39:07 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:07.264765 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.012s sys 0.000s I0918 20:39:07.266773 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:07.275876 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:39:07.276108 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-0-root uuid: "c099560cc34545d88ee0f5408bf0078e" format_stamp: "Formatted at 2019-09-18 20:39:07 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:07.276489 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:07.365658 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:37057 I0918 20:39:07.367508 19320 webserver.cc:374] Webserver started at http://127.18.222.62:44307/ using document root and password file I0918 20:39:07.369383 24644 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:07.374421 24644 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:07.385614 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 151092us I0918 20:39:07.386116 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:07.386355 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:07.386612 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:07.386775 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:07.388132 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-1-root/instance: uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" format_stamp: "Formatted at 2019-09-18 20:39:07 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:07.389291 24644 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:07.391206 24644 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:42813: Network error: Client connection negotiation failed: client connection to 127.18.222.61:42813: connect: Connection refused (error 111) I0918 20:39:07.398368 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.010s sys 0.000s I0918 20:39:07.400157 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:07.407958 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:07.408147 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-1-root uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" format_stamp: "Formatted at 2019-09-18 20:39:07 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:07.408421 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:07.426604 24644 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } attempt: 1 W0918 20:39:07.428828 24644 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:42813: Network error: Client connection negotiation failed: client connection to 127.18.222.61:42813: connect: Connection refused (error 111) I0918 20:39:07.481211 24644 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } attempt: 2 W0918 20:39:07.483356 24644 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:42813: Network error: Client connection negotiation failed: client connection to 127.18.222.61:42813: connect: Connection refused (error 111) I0918 20:39:07.492359 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:42813 I0918 20:39:07.494237 19320 webserver.cc:374] Webserver started at http://127.18.222.61:44771/ using document root and password file I0918 20:39:07.495935 24701 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:07.500701 24701 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:07.515682 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 151092us I0918 20:39:07.516156 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:07.516379 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:07.516639 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:07.516815 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:07.517539 24701 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:07.517951 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-2-root/instance: uuid: "1b0782873fac45c19186dd7e9966b568" format_stamp: "Formatted at 2019-09-18 20:39:07 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:07.530674 24701 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:07.530939 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.013s user 0.010s sys 0.005s W0918 20:39:07.532622 24701 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:45381: Network error: Client connection negotiation failed: client connection to 127.18.222.60:45381: connect: Connection refused (error 111) I0918 20:39:07.533217 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:07.541373 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:07.541574 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-2-root uuid: "1b0782873fac45c19186dd7e9966b568" format_stamp: "Formatted at 2019-09-18 20:39:07 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:07.541833 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:07.549763 24644 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } attempt: 3 I0918 20:39:07.557996 24701 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } attempt: 1 W0918 20:39:07.560223 24701 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:45381: Network error: Client connection negotiation failed: client connection to 127.18.222.60:45381: connect: Connection refused (error 111) I0918 20:39:07.561841 24644 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:07.563601 24644 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:45381: Network error: Client connection negotiation failed: client connection to 127.18.222.60:45381: connect: Connection refused (error 111) I0918 20:39:07.606312 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:45381 I0918 20:39:07.608196 19320 webserver.cc:374] Webserver started at http://127.18.222.60:35537/ using document root and password file I0918 20:39:07.609197 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0918 20:39:07.610111 24758 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:07.615022 24758 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:07.618058 24644 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } attempt: 1 I0918 20:39:07.628094 24758 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:07.629623 24701 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } attempt: 2 I0918 20:39:07.640123 24644 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e: Bootstrap starting. I0918 20:39:07.643182 24758 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:07.644466 24644 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e: No blocks or log segments found. Creating new log. I0918 20:39:07.650228 24701 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc: Bootstrap starting. I0918 20:39:07.653704 24701 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc: No blocks or log segments found. Creating new log. I0918 20:39:07.657886 24644 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e: No bootstrap required, opened a new log I0918 20:39:07.660334 24701 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc: No bootstrap required, opened a new log I0918 20:39:07.661697 24758 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568: Bootstrap starting. I0918 20:39:07.662015 24701 raft_consensus.cc:340] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:07.662539 24701 raft_consensus.cc:366] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:07.662724 24701 raft_consensus.cc:705] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ca09a1f9e5496f95857e93a1bfddfc, State: Initialized, Role: FOLLOWER I0918 20:39:07.663269 24701 consensus_queue.cc:229] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:07.659662 24644 raft_consensus.cc:340] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:07.664330 24644 raft_consensus.cc:366] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:07.664479 24758 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568: No blocks or log segments found. Creating new log. I0918 20:39:07.664604 24644 raft_consensus.cc:705] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c099560cc34545d88ee0f5408bf0078e, State: Initialized, Role: FOLLOWER I0918 20:39:07.665351 24644 consensus_queue.cc:229] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:07.666900 24701 sys_catalog.cc:340] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:07.667932 24701 sys_catalog.cc:343] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:07.667259 24764 sys_catalog.cc:340] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:07.669059 24764 sys_catalog.cc:343] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:07.668581 24644 sys_catalog.cc:340] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:07.669180 24701 sys_catalog.cc:428] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:07.669180 24758 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568: No bootstrap required, opened a new log I0918 20:39:07.669438 24644 sys_catalog.cc:343] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:07.668483 24766 sys_catalog.cc:340] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:07.670121 24766 sys_catalog.cc:343] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:07.671000 24644 sys_catalog.cc:428] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:07.672364 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 I0918 20:39:07.672735 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 I0918 20:39:07.672262 24758 raft_consensus.cc:340] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:07.673221 24758 raft_consensus.cc:366] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:07.673473 24758 raft_consensus.cc:705] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0782873fac45c19186dd7e9966b568, State: Initialized, Role: FOLLOWER W0918 20:39:07.673787 24767 catalog_manager.cc:1155] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:07.674043 24767 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:07.674309 24758 consensus_queue.cc:229] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } W0918 20:39:07.675107 24768 catalog_manager.cc:1155] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:07.675415 24768 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:07.676213 24758 sys_catalog.cc:340] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:07.676337 24769 sys_catalog.cc:340] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:07.676813 24758 sys_catalog.cc:343] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:07.676992 24769 sys_catalog.cc:343] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:07.677847 24758 sys_catalog.cc:428] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [sys.catalog]: configured and running, proceeding with master startup. W0918 20:39:07.681339 24770 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:07.681594 24770 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:07.695561 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 151092us I0918 20:39:07.695945 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:07.696094 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:07.696277 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:07.696394 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:07.697381 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/ts-0-root/instance: uuid: "c6071a1b02054de29eddfe6865eafb8e" format_stamp: "Formatted at 2019-09-18 20:39:07 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:07.707579 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.011s sys 0.000s I0918 20:39:07.709285 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:07.717584 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:39:07.717816 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/ts-0-root uuid: "c6071a1b02054de29eddfe6865eafb8e" format_stamp: "Formatted at 2019-09-18 20:39:07 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:07.718087 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:07.772370 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:07.772553 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:07.810515 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:45627 I0918 20:39:07.812337 19320 webserver.cc:374] Webserver started at http://127.18.222.1:35011/ using document root and password file I0918 20:39:07.832813 24835 heartbeater.cc:322] Connected to a master server at 127.18.222.60:45381 I0918 20:39:07.833142 24835 heartbeater.cc:406] Registering TS with master... I0918 20:39:07.833631 24835 heartbeater.cc:455] Master 127.18.222.60:45381 requested a full tablet report, sending... I0918 20:39:07.835686 24725 ts_manager.cc:186] Registered new tserver with Master: c6071a1b02054de29eddfe6865eafb8e (127.18.222.1:45627) I0918 20:39:07.840495 24836 heartbeater.cc:322] Connected to a master server at 127.18.222.61:42813 I0918 20:39:07.840770 24836 heartbeater.cc:406] Registering TS with master... I0918 20:39:07.841207 24836 heartbeater.cc:455] Master 127.18.222.61:42813 requested a full tablet report, sending... I0918 20:39:07.843940 24668 ts_manager.cc:186] Registered new tserver with Master: c6071a1b02054de29eddfe6865eafb8e (127.18.222.1:45627) I0918 20:39:07.844678 24837 heartbeater.cc:322] Connected to a master server at 127.18.222.62:37057 I0918 20:39:07.845140 24837 heartbeater.cc:406] Registering TS with master... I0918 20:39:07.845602 24837 heartbeater.cc:455] Master 127.18.222.62:37057 requested a full tablet report, sending... I0918 20:39:07.846187 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 151092us I0918 20:39:07.846555 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:07.846735 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:07.846967 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:07.847106 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:07.847949 24611 ts_manager.cc:186] Registered new tserver with Master: c6071a1b02054de29eddfe6865eafb8e (127.18.222.1:45627) I0918 20:39:07.848404 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/ts-1-root/instance: uuid: "20d7112efcfa495c9baa82bd089f4928" format_stamp: "Formatted at 2019-09-18 20:39:07 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:07.859856 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.011s sys 0.001s I0918 20:39:07.862087 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:07.870692 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:07.870929 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/ts-1-root uuid: "20d7112efcfa495c9baa82bd089f4928" format_stamp: "Formatted at 2019-09-18 20:39:07 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:07.871212 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:07.909279 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:07.909574 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:07.920151 24766 raft_consensus.cc:472] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:07.920755 24766 raft_consensus.cc:494] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:07.922849 24766 leader_election.cc:296] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6ca09a1f9e5496f95857e93a1bfddfc (127.18.222.61:42813), 1b0782873fac45c19186dd7e9966b568 (127.18.222.60:45381) I0918 20:39:07.924178 24735 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c099560cc34545d88ee0f5408bf0078e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0782873fac45c19186dd7e9966b568" is_pre_election: true I0918 20:39:07.924697 24678 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c099560cc34545d88ee0f5408bf0078e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" is_pre_election: true I0918 20:39:07.925417 24678 raft_consensus.cc:2350] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c099560cc34545d88ee0f5408bf0078e in term 0. I0918 20:39:07.926506 24735 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c099560cc34545d88ee0f5408bf0078e in term 0. I0918 20:39:07.927875 24592 leader_election.cc:310] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0782873fac45c19186dd7e9966b568, c099560cc34545d88ee0f5408bf0078e; no voters: I0918 20:39:07.929888 24766 raft_consensus.cc:2644] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:07.930197 24766 raft_consensus.cc:472] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:07.930451 24766 raft_consensus.cc:2890] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:07.934725 24766 raft_consensus.cc:494] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:07.937911 24766 leader_election.cc:296] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [CANDIDATE]: Term 1 election: Requested vote from peers b6ca09a1f9e5496f95857e93a1bfddfc (127.18.222.61:42813), 1b0782873fac45c19186dd7e9966b568 (127.18.222.60:45381) I0918 20:39:07.938112 24735 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c099560cc34545d88ee0f5408bf0078e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0782873fac45c19186dd7e9966b568" I0918 20:39:07.938271 24678 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c099560cc34545d88ee0f5408bf0078e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" I0918 20:39:07.938818 24735 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:07.938951 24678 raft_consensus.cc:2890] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:07.944460 24735 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c099560cc34545d88ee0f5408bf0078e in term 1. I0918 20:39:07.945681 24592 leader_election.cc:310] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0782873fac45c19186dd7e9966b568, c099560cc34545d88ee0f5408bf0078e; no voters: I0918 20:39:07.946491 24678 raft_consensus.cc:2350] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c099560cc34545d88ee0f5408bf0078e in term 1. I0918 20:39:07.946719 24766 raft_consensus.cc:2644] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:07.954774 24766 raft_consensus.cc:667] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 1 LEADER]: Becoming Leader. State: Replica: c099560cc34545d88ee0f5408bf0078e, State: Running, Role: LEADER I0918 20:39:07.956118 24766 consensus_queue.cc:206] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:07.961081 24865 sys_catalog.cc:340] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c099560cc34545d88ee0f5408bf0078e. Latest consensus state: current_term: 1 leader_uuid: "c099560cc34545d88ee0f5408bf0078e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:07.962009 24865 sys_catalog.cc:343] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [sys.catalog]: This master's current role is: LEADER I0918 20:39:07.964468 24873 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:39:07.969072 24873 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:39:08.004168 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.2:42835 I0918 20:39:08.008350 19320 webserver.cc:374] Webserver started at http://127.18.222.2:41919/ using document root and password file I0918 20:39:08.044044 24909 heartbeater.cc:322] Connected to a master server at 127.18.222.60:45381 I0918 20:39:08.044420 24909 heartbeater.cc:406] Registering TS with master... I0918 20:39:08.044978 24909 heartbeater.cc:455] Master 127.18.222.60:45381 requested a full tablet report, sending... I0918 20:39:08.045526 24910 heartbeater.cc:322] Connected to a master server at 127.18.222.61:42813 I0918 20:39:08.045931 24910 heartbeater.cc:406] Registering TS with master... I0918 20:39:08.046464 24910 heartbeater.cc:455] Master 127.18.222.61:42813 requested a full tablet report, sending... I0918 20:39:08.047794 24725 ts_manager.cc:186] Registered new tserver with Master: 20d7112efcfa495c9baa82bd089f4928 (127.18.222.2:42835) I0918 20:39:08.051790 24668 ts_manager.cc:186] Registered new tserver with Master: 20d7112efcfa495c9baa82bd089f4928 (127.18.222.2:42835) I0918 20:39:08.055596 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 151592us I0918 20:39:08.054926 24735 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 1 FOLLOWER]: Refusing update from remote peer c099560cc34545d88ee0f5408bf0078e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:08.056272 24678 raft_consensus.cc:1184] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 1 FOLLOWER]: Refusing update from remote peer c099560cc34545d88ee0f5408bf0078e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:08.056403 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:08.056728 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:08.057058 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:08.057246 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:08.057704 24865 consensus_queue.cc:984] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:08.058410 24766 consensus_queue.cc:984] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:08.058676 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/ts-2-root/instance: uuid: "4f96668e49f44cfe9cdfe6462347a594" format_stamp: "Formatted at 2019-09-18 20:39:08 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:08.062700 24913 heartbeater.cc:322] Connected to a master server at 127.18.222.62:37057 I0918 20:39:08.063071 24913 heartbeater.cc:406] Registering TS with master... I0918 20:39:08.063752 24913 heartbeater.cc:455] Master 127.18.222.62:37057 requested a full tablet report, sending... I0918 20:39:08.065640 24611 ts_manager.cc:186] Registered new tserver with Master: 20d7112efcfa495c9baa82bd089f4928 (127.18.222.2:42835) I0918 20:39:08.074390 24769 sys_catalog.cc:340] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c099560cc34545d88ee0f5408bf0078e. Latest consensus state: current_term: 1 leader_uuid: "c099560cc34545d88ee0f5408bf0078e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:08.075330 24769 sys_catalog.cc:343] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:08.082422 24764 sys_catalog.cc:340] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c099560cc34545d88ee0f5408bf0078e. Latest consensus state: current_term: 1 leader_uuid: "c099560cc34545d88ee0f5408bf0078e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:08.084302 24764 sys_catalog.cc:343] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:08.099633 24865 sys_catalog.cc:340] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c099560cc34545d88ee0f5408bf0078e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:08.100625 24865 sys_catalog.cc:343] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [sys.catalog]: This master's current role is: LEADER I0918 20:39:08.102658 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.043s user 0.018s sys 0.009s I0918 20:39:08.103332 24865 sys_catalog.cc:340] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c099560cc34545d88ee0f5408bf0078e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:08.104346 24865 sys_catalog.cc:343] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [sys.catalog]: This master's current role is: LEADER I0918 20:39:08.103994 24764 sys_catalog.cc:340] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c099560cc34545d88ee0f5408bf0078e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:08.105127 24764 sys_catalog.cc:343] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:08.105013 24873 catalog_manager.cc:981] Generated new certificate authority record I0918 20:39:08.107147 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:08.109834 24873 catalog_manager.cc:1106] Loading token signing keys... I0918 20:39:08.112484 24769 sys_catalog.cc:340] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c099560cc34545d88ee0f5408bf0078e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:08.113334 24769 sys_catalog.cc:343] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:08.129410 24873 catalog_manager.cc:4446] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e: Generated new TSK 0 I0918 20:39:08.130136 24873 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:39:08.135206 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0918 20:39:08.135589 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/ts-2-root uuid: "4f96668e49f44cfe9cdfe6462347a594" format_stamp: "Formatted at 2019-09-18 20:39:08 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:08.135993 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:08.141743 24610 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39358 I0918 20:39:08.151937 24611 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39352 I0918 20:39:08.161306 24913 heartbeater.cc:447] Master 127.18.222.62:37057 was elected leader, sending a full tablet report... I0918 20:39:08.164512 24837 heartbeater.cc:447] Master 127.18.222.62:37057 was elected leader, sending a full tablet report... I0918 20:39:08.168332 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:08.168635 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:08.233866 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.3:45255 I0918 20:39:08.235760 19320 webserver.cc:374] Webserver started at http://127.18.222.3:37251/ using document root and password file I0918 20:39:08.271754 24989 heartbeater.cc:322] Connected to a master server at 127.18.222.61:42813 I0918 20:39:08.272189 24989 heartbeater.cc:406] Registering TS with master... I0918 20:39:08.273219 24989 heartbeater.cc:455] Master 127.18.222.61:42813 requested a full tablet report, sending... I0918 20:39:08.273396 24988 heartbeater.cc:322] Connected to a master server at 127.18.222.60:45381 I0918 20:39:08.273808 24988 heartbeater.cc:406] Registering TS with master... I0918 20:39:08.274332 24988 heartbeater.cc:455] Master 127.18.222.60:45381 requested a full tablet report, sending... I0918 20:39:08.276322 24668 ts_manager.cc:186] Registered new tserver with Master: 4f96668e49f44cfe9cdfe6462347a594 (127.18.222.3:45255) I0918 20:39:08.276798 24723 ts_manager.cc:186] Registered new tserver with Master: 4f96668e49f44cfe9cdfe6462347a594 (127.18.222.3:45255) I0918 20:39:08.282829 24992 heartbeater.cc:322] Connected to a master server at 127.18.222.62:37057 I0918 20:39:08.283097 24992 heartbeater.cc:406] Registering TS with master... I0918 20:39:08.283542 24992 heartbeater.cc:455] Master 127.18.222.62:37057 requested a full tablet report, sending... I0918 20:39:08.285063 24611 ts_manager.cc:186] Registered new tserver with Master: 4f96668e49f44cfe9cdfe6462347a594 (127.18.222.3:45255) I0918 20:39:08.285513 19320 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.028722253s I0918 20:39:08.286532 24611 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39364 I0918 20:39:08.307850 24992 heartbeater.cc:447] Master 127.18.222.62:37057 was elected leader, sending a full tablet report... I0918 20:39:08.354391 24611 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39366: name: "ksck-test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\n\000\000\000\004\001\024\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:39:08.357051 24611 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:39:08.477107 24955 tablet_service.cc:1022] Processing CreateTablet for tablet 98003d0b22fb4a248daa8ba50ca38265 (table=ksck-test-table [id=7aa24a7d75d54b7889f05c617afd1dfb]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:08.478240 24955 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 98003d0b22fb4a248daa8ba50ca38265. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:08.488485 24801 tablet_service.cc:1022] Processing CreateTablet for tablet 8119ec3019aa43a78aace2e7ed85055d (table=ksck-test-table [id=7aa24a7d75d54b7889f05c617afd1dfb]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:08.488873 24954 tablet_service.cc:1022] Processing CreateTablet for tablet 8119ec3019aa43a78aace2e7ed85055d (table=ksck-test-table [id=7aa24a7d75d54b7889f05c617afd1dfb]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:08.489223 24953 tablet_service.cc:1022] Processing CreateTablet for tablet 4b86906ac1084018a23f202664ca0276 (table=ksck-test-table [id=7aa24a7d75d54b7889f05c617afd1dfb]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:08.489707 24801 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 8119ec3019aa43a78aace2e7ed85055d. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:08.489866 24954 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 8119ec3019aa43a78aace2e7ed85055d. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:08.490383 24953 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 4b86906ac1084018a23f202664ca0276. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:08.492504 24802 tablet_service.cc:1022] Processing CreateTablet for tablet 98003d0b22fb4a248daa8ba50ca38265 (table=ksck-test-table [id=7aa24a7d75d54b7889f05c617afd1dfb]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:08.493470 24802 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 98003d0b22fb4a248daa8ba50ca38265. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:08.496102 24800 tablet_service.cc:1022] Processing CreateTablet for tablet 4b86906ac1084018a23f202664ca0276 (table=ksck-test-table [id=7aa24a7d75d54b7889f05c617afd1dfb]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:08.497197 24800 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 4b86906ac1084018a23f202664ca0276. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:08.530584 25009 tablet_bootstrap.cc:439] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594: Bootstrap starting. I0918 20:39:08.540195 25009 tablet_bootstrap.cc:590] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594: No blocks or log segments found. Creating new log. I0918 20:39:08.545182 24876 tablet_service.cc:1022] Processing CreateTablet for tablet 98003d0b22fb4a248daa8ba50ca38265 (table=ksck-test-table [id=7aa24a7d75d54b7889f05c617afd1dfb]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:08.545786 24875 tablet_service.cc:1022] Processing CreateTablet for tablet 8119ec3019aa43a78aace2e7ed85055d (table=ksck-test-table [id=7aa24a7d75d54b7889f05c617afd1dfb]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:08.546249 24876 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 98003d0b22fb4a248daa8ba50ca38265. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:08.546753 24875 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 8119ec3019aa43a78aace2e7ed85055d. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:08.559641 24874 tablet_service.cc:1022] Processing CreateTablet for tablet 4b86906ac1084018a23f202664ca0276 (table=ksck-test-table [id=7aa24a7d75d54b7889f05c617afd1dfb]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:08.561424 24874 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 4b86906ac1084018a23f202664ca0276. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:08.586614 25010 tablet_bootstrap.cc:439] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e: Bootstrap starting. I0918 20:39:08.592731 25012 tablet_bootstrap.cc:439] T 8119ec3019aa43a78aace2e7ed85055d P 20d7112efcfa495c9baa82bd089f4928: Bootstrap starting. I0918 20:39:08.596871 25009 tablet_bootstrap.cc:439] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594: No bootstrap required, opened a new log I0918 20:39:08.597595 25009 ts_tablet_manager.cc:1085] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594: Time spent bootstrapping tablet: real 0.067s user 0.012s sys 0.008s I0918 20:39:08.599052 25012 tablet_bootstrap.cc:590] T 8119ec3019aa43a78aace2e7ed85055d P 20d7112efcfa495c9baa82bd089f4928: No blocks or log segments found. Creating new log. I0918 20:39:08.599773 25009 raft_consensus.cc:340] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } I0918 20:39:08.600708 25009 raft_consensus.cc:366] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:08.600939 25009 raft_consensus.cc:705] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f96668e49f44cfe9cdfe6462347a594, State: Initialized, Role: FOLLOWER I0918 20:39:08.601346 25010 tablet_bootstrap.cc:590] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e: No blocks or log segments found. Creating new log. I0918 20:39:08.601724 25009 consensus_queue.cc:229] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } I0918 20:39:08.639520 25009 ts_tablet_manager.cc:1113] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594: Time spent starting tablet: real 0.042s user 0.029s sys 0.008s I0918 20:39:08.640460 25009 tablet_bootstrap.cc:439] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594: Bootstrap starting. I0918 20:39:08.645757 25009 tablet_bootstrap.cc:590] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594: No blocks or log segments found. Creating new log. I0918 20:39:08.662453 25010 tablet_bootstrap.cc:439] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e: No bootstrap required, opened a new log I0918 20:39:08.663080 25010 ts_tablet_manager.cc:1085] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e: Time spent bootstrapping tablet: real 0.077s user 0.022s sys 0.017s I0918 20:39:08.665537 25010 raft_consensus.cc:340] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } I0918 20:39:08.666376 25010 raft_consensus.cc:366] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:08.666640 25010 raft_consensus.cc:705] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6071a1b02054de29eddfe6865eafb8e, State: Initialized, Role: FOLLOWER I0918 20:39:08.667464 25010 consensus_queue.cc:229] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } I0918 20:39:08.697175 25010 ts_tablet_manager.cc:1113] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e: Time spent starting tablet: real 0.034s user 0.009s sys 0.005s I0918 20:39:08.697607 24770 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568: acquiring CA information for follower catalog manager: success I0918 20:39:08.698025 25010 tablet_bootstrap.cc:439] T 4b86906ac1084018a23f202664ca0276 P c6071a1b02054de29eddfe6865eafb8e: Bootstrap starting. I0918 20:39:08.702601 25012 tablet_bootstrap.cc:439] T 8119ec3019aa43a78aace2e7ed85055d P 20d7112efcfa495c9baa82bd089f4928: No bootstrap required, opened a new log I0918 20:39:08.704246 25012 ts_tablet_manager.cc:1085] T 8119ec3019aa43a78aace2e7ed85055d P 20d7112efcfa495c9baa82bd089f4928: Time spent bootstrapping tablet: real 0.112s user 0.011s sys 0.018s I0918 20:39:08.706281 25010 tablet_bootstrap.cc:590] T 4b86906ac1084018a23f202664ca0276 P c6071a1b02054de29eddfe6865eafb8e: No blocks or log segments found. Creating new log. I0918 20:39:08.707926 24767 catalog_manager.cc:1153] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc: acquiring CA information for follower catalog manager: success I0918 20:39:08.715976 24767 catalog_manager.cc:1181] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:08.716085 24770 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:08.715574 25012 raft_consensus.cc:340] T 8119ec3019aa43a78aace2e7ed85055d P 20d7112efcfa495c9baa82bd089f4928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } I0918 20:39:08.717432 25012 raft_consensus.cc:366] T 8119ec3019aa43a78aace2e7ed85055d P 20d7112efcfa495c9baa82bd089f4928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:08.717689 25012 raft_consensus.cc:705] T 8119ec3019aa43a78aace2e7ed85055d P 20d7112efcfa495c9baa82bd089f4928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d7112efcfa495c9baa82bd089f4928, State: Initialized, Role: FOLLOWER I0918 20:39:08.718338 25012 consensus_queue.cc:229] T 8119ec3019aa43a78aace2e7ed85055d P 20d7112efcfa495c9baa82bd089f4928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } I0918 20:39:08.728241 25018 raft_consensus.cc:472] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:08.729122 25009 tablet_bootstrap.cc:439] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594: No bootstrap required, opened a new log I0918 20:39:08.730576 25009 ts_tablet_manager.cc:1085] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594: Time spent bootstrapping tablet: real 0.090s user 0.014s sys 0.019s I0918 20:39:08.730419 25018 raft_consensus.cc:494] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } I0918 20:39:08.733086 25009 raft_consensus.cc:340] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } I0918 20:39:08.733896 25009 raft_consensus.cc:366] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:08.734128 25009 raft_consensus.cc:705] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f96668e49f44cfe9cdfe6462347a594, State: Initialized, Role: FOLLOWER I0918 20:39:08.734907 25009 consensus_queue.cc:229] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } I0918 20:39:08.737776 25009 ts_tablet_manager.cc:1113] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s I0918 20:39:08.729933 25010 tablet_bootstrap.cc:439] T 4b86906ac1084018a23f202664ca0276 P c6071a1b02054de29eddfe6865eafb8e: No bootstrap required, opened a new log I0918 20:39:08.739143 25010 ts_tablet_manager.cc:1085] T 4b86906ac1084018a23f202664ca0276 P c6071a1b02054de29eddfe6865eafb8e: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.013s I0918 20:39:08.741423 25012 ts_tablet_manager.cc:1113] T 8119ec3019aa43a78aace2e7ed85055d P 20d7112efcfa495c9baa82bd089f4928: Time spent starting tablet: real 0.036s user 0.009s sys 0.003s I0918 20:39:08.742561 25010 raft_consensus.cc:340] T 4b86906ac1084018a23f202664ca0276 P c6071a1b02054de29eddfe6865eafb8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } I0918 20:39:08.743330 25010 raft_consensus.cc:366] T 4b86906ac1084018a23f202664ca0276 P c6071a1b02054de29eddfe6865eafb8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:08.743546 25010 raft_consensus.cc:705] T 4b86906ac1084018a23f202664ca0276 P c6071a1b02054de29eddfe6865eafb8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6071a1b02054de29eddfe6865eafb8e, State: Initialized, Role: FOLLOWER I0918 20:39:08.744240 25010 consensus_queue.cc:229] T 4b86906ac1084018a23f202664ca0276 P c6071a1b02054de29eddfe6865eafb8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } I0918 20:39:08.746006 25010 ts_tablet_manager.cc:1113] T 4b86906ac1084018a23f202664ca0276 P c6071a1b02054de29eddfe6865eafb8e: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s I0918 20:39:08.746773 25010 tablet_bootstrap.cc:439] T 98003d0b22fb4a248daa8ba50ca38265 P c6071a1b02054de29eddfe6865eafb8e: Bootstrap starting. I0918 20:39:08.751566 25010 tablet_bootstrap.cc:590] T 98003d0b22fb4a248daa8ba50ca38265 P c6071a1b02054de29eddfe6865eafb8e: No blocks or log segments found. Creating new log. I0918 20:39:08.774431 25012 tablet_bootstrap.cc:439] T 98003d0b22fb4a248daa8ba50ca38265 P 20d7112efcfa495c9baa82bd089f4928: Bootstrap starting. I0918 20:39:08.778261 25012 tablet_bootstrap.cc:590] T 98003d0b22fb4a248daa8ba50ca38265 P 20d7112efcfa495c9baa82bd089f4928: No blocks or log segments found. Creating new log. I0918 20:39:08.797475 25014 raft_consensus.cc:472] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:08.798107 25014 raft_consensus.cc:494] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } I0918 20:39:08.803681 25014 leader_election.cc:296] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6071a1b02054de29eddfe6865eafb8e (127.18.222.1:45627), 20d7112efcfa495c9baa82bd089f4928 (127.18.222.2:42835) I0918 20:39:08.808425 25009 tablet_bootstrap.cc:439] T 8119ec3019aa43a78aace2e7ed85055d P 4f96668e49f44cfe9cdfe6462347a594: Bootstrap starting. I0918 20:39:08.810706 25018 leader_election.cc:296] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20d7112efcfa495c9baa82bd089f4928 (127.18.222.2:42835), 4f96668e49f44cfe9cdfe6462347a594 (127.18.222.3:45255) I0918 20:39:08.812983 25010 tablet_bootstrap.cc:439] T 98003d0b22fb4a248daa8ba50ca38265 P c6071a1b02054de29eddfe6865eafb8e: No bootstrap required, opened a new log I0918 20:39:08.832841 25009 tablet_bootstrap.cc:590] T 8119ec3019aa43a78aace2e7ed85055d P 4f96668e49f44cfe9cdfe6462347a594: No blocks or log segments found. Creating new log. I0918 20:39:08.833016 25012 tablet_bootstrap.cc:439] T 98003d0b22fb4a248daa8ba50ca38265 P 20d7112efcfa495c9baa82bd089f4928: No bootstrap required, opened a new log I0918 20:39:08.831912 25010 ts_tablet_manager.cc:1085] T 98003d0b22fb4a248daa8ba50ca38265 P c6071a1b02054de29eddfe6865eafb8e: Time spent bootstrapping tablet: real 0.084s user 0.033s sys 0.022s I0918 20:39:08.835861 25012 ts_tablet_manager.cc:1085] T 98003d0b22fb4a248daa8ba50ca38265 P 20d7112efcfa495c9baa82bd089f4928: Time spent bootstrapping tablet: real 0.062s user 0.015s sys 0.034s I0918 20:39:08.874188 25010 raft_consensus.cc:340] T 98003d0b22fb4a248daa8ba50ca38265 P c6071a1b02054de29eddfe6865eafb8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } I0918 20:39:08.875026 25010 raft_consensus.cc:366] T 98003d0b22fb4a248daa8ba50ca38265 P c6071a1b02054de29eddfe6865eafb8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:08.875277 25010 raft_consensus.cc:705] T 98003d0b22fb4a248daa8ba50ca38265 P c6071a1b02054de29eddfe6865eafb8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6071a1b02054de29eddfe6865eafb8e, State: Initialized, Role: FOLLOWER I0918 20:39:08.881801 25010 consensus_queue.cc:229] T 98003d0b22fb4a248daa8ba50ca38265 P c6071a1b02054de29eddfe6865eafb8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } I0918 20:39:08.887660 25012 raft_consensus.cc:340] T 98003d0b22fb4a248daa8ba50ca38265 P 20d7112efcfa495c9baa82bd089f4928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } I0918 20:39:08.907065 25009 tablet_bootstrap.cc:439] T 8119ec3019aa43a78aace2e7ed85055d P 4f96668e49f44cfe9cdfe6462347a594: No bootstrap required, opened a new log I0918 20:39:08.909612 25009 ts_tablet_manager.cc:1085] T 8119ec3019aa43a78aace2e7ed85055d P 4f96668e49f44cfe9cdfe6462347a594: Time spent bootstrapping tablet: real 0.128s user 0.012s sys 0.018s I0918 20:39:08.906258 25010 ts_tablet_manager.cc:1113] T 98003d0b22fb4a248daa8ba50ca38265 P c6071a1b02054de29eddfe6865eafb8e: Time spent starting tablet: real 0.070s user 0.006s sys 0.001s I0918 20:39:08.907944 25012 raft_consensus.cc:366] T 98003d0b22fb4a248daa8ba50ca38265 P 20d7112efcfa495c9baa82bd089f4928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:08.911486 25012 raft_consensus.cc:705] T 98003d0b22fb4a248daa8ba50ca38265 P 20d7112efcfa495c9baa82bd089f4928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d7112efcfa495c9baa82bd089f4928, State: Initialized, Role: FOLLOWER I0918 20:39:08.922214 25014 raft_consensus.cc:472] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:08.925213 25014 raft_consensus.cc:494] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } I0918 20:39:08.927505 25009 raft_consensus.cc:340] T 8119ec3019aa43a78aace2e7ed85055d P 4f96668e49f44cfe9cdfe6462347a594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } I0918 20:39:08.928417 25009 raft_consensus.cc:366] T 8119ec3019aa43a78aace2e7ed85055d P 4f96668e49f44cfe9cdfe6462347a594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:08.928645 25009 raft_consensus.cc:705] T 8119ec3019aa43a78aace2e7ed85055d P 4f96668e49f44cfe9cdfe6462347a594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f96668e49f44cfe9cdfe6462347a594, State: Initialized, Role: FOLLOWER I0918 20:39:08.929277 25009 consensus_queue.cc:229] T 8119ec3019aa43a78aace2e7ed85055d P 4f96668e49f44cfe9cdfe6462347a594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } I0918 20:39:08.927536 25014 leader_election.cc:296] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6071a1b02054de29eddfe6865eafb8e (127.18.222.1:45627), 20d7112efcfa495c9baa82bd089f4928 (127.18.222.2:42835) I0918 20:39:08.930950 25012 consensus_queue.cc:229] T 98003d0b22fb4a248daa8ba50ca38265 P 20d7112efcfa495c9baa82bd089f4928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } I0918 20:39:08.941587 24886 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8119ec3019aa43a78aace2e7ed85055d" candidate_uuid: "c6071a1b02054de29eddfe6865eafb8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d7112efcfa495c9baa82bd089f4928" is_pre_election: true I0918 20:39:08.942392 24886 raft_consensus.cc:2350] T 8119ec3019aa43a78aace2e7ed85055d P 20d7112efcfa495c9baa82bd089f4928 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6071a1b02054de29eddfe6865eafb8e in term 0. I0918 20:39:08.947824 24775 leader_election.cc:310] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20d7112efcfa495c9baa82bd089f4928, c6071a1b02054de29eddfe6865eafb8e; no voters: I0918 20:39:08.948763 25018 raft_consensus.cc:2644] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:08.949055 25018 raft_consensus.cc:472] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:08.949355 25018 raft_consensus.cc:2890] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:08.957005 24965 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8119ec3019aa43a78aace2e7ed85055d" candidate_uuid: "c6071a1b02054de29eddfe6865eafb8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f96668e49f44cfe9cdfe6462347a594" is_pre_election: true I0918 20:39:08.957999 24965 raft_consensus.cc:2350] T 8119ec3019aa43a78aace2e7ed85055d P 4f96668e49f44cfe9cdfe6462347a594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6071a1b02054de29eddfe6865eafb8e in term 0. I0918 20:39:08.964491 25018 raft_consensus.cc:494] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } I0918 20:39:08.967032 24886 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8119ec3019aa43a78aace2e7ed85055d" candidate_uuid: "c6071a1b02054de29eddfe6865eafb8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d7112efcfa495c9baa82bd089f4928" I0918 20:39:08.967587 24886 raft_consensus.cc:2890] T 8119ec3019aa43a78aace2e7ed85055d P 20d7112efcfa495c9baa82bd089f4928 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:09.026726 24886 raft_consensus.cc:2350] T 8119ec3019aa43a78aace2e7ed85055d P 20d7112efcfa495c9baa82bd089f4928 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6071a1b02054de29eddfe6865eafb8e in term 1. I0918 20:39:09.027988 24775 leader_election.cc:310] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20d7112efcfa495c9baa82bd089f4928, c6071a1b02054de29eddfe6865eafb8e; no voters: I0918 20:39:09.028954 25017 raft_consensus.cc:2644] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:09.038507 25018 leader_election.cc:296] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e [CANDIDATE]: Term 1 election: Requested vote from peers 20d7112efcfa495c9baa82bd089f4928 (127.18.222.2:42835), 4f96668e49f44cfe9cdfe6462347a594 (127.18.222.3:45255) I0918 20:39:09.041963 25012 ts_tablet_manager.cc:1113] T 98003d0b22fb4a248daa8ba50ca38265 P 20d7112efcfa495c9baa82bd089f4928: Time spent starting tablet: real 0.201s user 0.008s sys 0.000s I0918 20:39:09.042873 25012 tablet_bootstrap.cc:439] T 4b86906ac1084018a23f202664ca0276 P 20d7112efcfa495c9baa82bd089f4928: Bootstrap starting. I0918 20:39:09.039455 24965 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8119ec3019aa43a78aace2e7ed85055d" candidate_uuid: "c6071a1b02054de29eddfe6865eafb8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f96668e49f44cfe9cdfe6462347a594" I0918 20:39:09.045941 24965 raft_consensus.cc:2890] T 8119ec3019aa43a78aace2e7ed85055d P 4f96668e49f44cfe9cdfe6462347a594 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:09.053419 24965 raft_consensus.cc:2350] T 8119ec3019aa43a78aace2e7ed85055d P 4f96668e49f44cfe9cdfe6462347a594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6071a1b02054de29eddfe6865eafb8e in term 1. I0918 20:39:09.055805 24812 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "4b86906ac1084018a23f202664ca0276" candidate_uuid: "4f96668e49f44cfe9cdfe6462347a594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6071a1b02054de29eddfe6865eafb8e" is_pre_election: true I0918 20:39:09.056138 24811 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "98003d0b22fb4a248daa8ba50ca38265" candidate_uuid: "4f96668e49f44cfe9cdfe6462347a594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6071a1b02054de29eddfe6865eafb8e" is_pre_election: true I0918 20:39:09.056638 24812 raft_consensus.cc:2350] T 4b86906ac1084018a23f202664ca0276 P c6071a1b02054de29eddfe6865eafb8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f96668e49f44cfe9cdfe6462347a594 in term 0. I0918 20:39:09.057866 24928 leader_election.cc:310] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f96668e49f44cfe9cdfe6462347a594, c6071a1b02054de29eddfe6865eafb8e; no voters: I0918 20:39:09.058856 24811 raft_consensus.cc:2350] T 98003d0b22fb4a248daa8ba50ca38265 P c6071a1b02054de29eddfe6865eafb8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f96668e49f44cfe9cdfe6462347a594 in term 0. I0918 20:39:09.059969 25029 raft_consensus.cc:2644] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:09.060120 24928 leader_election.cc:310] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f96668e49f44cfe9cdfe6462347a594, c6071a1b02054de29eddfe6865eafb8e; no voters: I0918 20:39:09.061240 25014 raft_consensus.cc:2644] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:09.061620 25014 raft_consensus.cc:472] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:09.061970 25014 raft_consensus.cc:2890] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 [term 0 FOLLOWER]: Advancing to term 1 W0918 20:39:09.044180 24845 outbound_call.cc:287] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.18.222.62:37057, user_credentials={real_user=slave}} blocked reactor thread for 59304.5us I0918 20:39:09.060482 25029 raft_consensus.cc:472] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:09.069195 25029 raft_consensus.cc:2890] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:09.062623 25017 raft_consensus.cc:667] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e [term 1 LEADER]: Becoming Leader. State: Replica: c6071a1b02054de29eddfe6865eafb8e, State: Running, Role: LEADER I0918 20:39:09.069998 24885 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "98003d0b22fb4a248daa8ba50ca38265" candidate_uuid: "4f96668e49f44cfe9cdfe6462347a594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d7112efcfa495c9baa82bd089f4928" is_pre_election: true I0918 20:39:09.072199 25017 consensus_queue.cc:206] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } I0918 20:39:09.072187 24885 raft_consensus.cc:2350] T 98003d0b22fb4a248daa8ba50ca38265 P 20d7112efcfa495c9baa82bd089f4928 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f96668e49f44cfe9cdfe6462347a594 in term 0. I0918 20:39:09.074646 24886 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "4b86906ac1084018a23f202664ca0276" candidate_uuid: "4f96668e49f44cfe9cdfe6462347a594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d7112efcfa495c9baa82bd089f4928" is_pre_election: true W0918 20:39:09.075932 24928 leader_election.cc:348] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 20d7112efcfa495c9baa82bd089f4928 (127.18.222.2:42835): Illegal state: must be running to vote when last-logged opid is not known I0918 20:39:09.069900 25014 raft_consensus.cc:494] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } I0918 20:39:09.077854 25009 ts_tablet_manager.cc:1113] T 8119ec3019aa43a78aace2e7ed85055d P 4f96668e49f44cfe9cdfe6462347a594: Time spent starting tablet: real 0.162s user 0.047s sys 0.038s I0918 20:39:09.110342 24610 catalog_manager.cc:4131] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e reported cstate change: term changed from 0 to 1, leader changed from to c6071a1b02054de29eddfe6865eafb8e (127.18.222.1). New cstate: current_term: 1 leader_uuid: "c6071a1b02054de29eddfe6865eafb8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } health_report { overall_health: UNKNOWN } } } I0918 20:39:09.114878 24811 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "98003d0b22fb4a248daa8ba50ca38265" candidate_uuid: "4f96668e49f44cfe9cdfe6462347a594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6071a1b02054de29eddfe6865eafb8e" I0918 20:39:09.115782 25014 leader_election.cc:296] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 [CANDIDATE]: Term 1 election: Requested vote from peers c6071a1b02054de29eddfe6865eafb8e (127.18.222.1:45627), 20d7112efcfa495c9baa82bd089f4928 (127.18.222.2:42835) I0918 20:39:09.115622 24811 raft_consensus.cc:2890] T 98003d0b22fb4a248daa8ba50ca38265 P c6071a1b02054de29eddfe6865eafb8e [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:09.118214 25029 raft_consensus.cc:494] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } I0918 20:39:09.121965 24812 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "4b86906ac1084018a23f202664ca0276" candidate_uuid: "4f96668e49f44cfe9cdfe6462347a594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6071a1b02054de29eddfe6865eafb8e" I0918 20:39:09.122598 24812 raft_consensus.cc:2890] T 4b86906ac1084018a23f202664ca0276 P c6071a1b02054de29eddfe6865eafb8e [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:09.127635 24812 raft_consensus.cc:2350] T 4b86906ac1084018a23f202664ca0276 P c6071a1b02054de29eddfe6865eafb8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f96668e49f44cfe9cdfe6462347a594 in term 1. I0918 20:39:09.128952 24928 leader_election.cc:310] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f96668e49f44cfe9cdfe6462347a594, c6071a1b02054de29eddfe6865eafb8e; no voters: I0918 20:39:09.129679 25014 raft_consensus.cc:2644] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:09.132112 24886 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "98003d0b22fb4a248daa8ba50ca38265" candidate_uuid: "4f96668e49f44cfe9cdfe6462347a594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d7112efcfa495c9baa82bd089f4928" I0918 20:39:09.132975 24886 raft_consensus.cc:2890] T 98003d0b22fb4a248daa8ba50ca38265 P 20d7112efcfa495c9baa82bd089f4928 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:09.138275 24811 raft_consensus.cc:2350] T 98003d0b22fb4a248daa8ba50ca38265 P c6071a1b02054de29eddfe6865eafb8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f96668e49f44cfe9cdfe6462347a594 in term 1. I0918 20:39:09.140118 24928 leader_election.cc:310] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f96668e49f44cfe9cdfe6462347a594, c6071a1b02054de29eddfe6865eafb8e; no voters: I0918 20:39:09.144047 25029 leader_election.cc:296] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [CANDIDATE]: Term 1 election: Requested vote from peers c6071a1b02054de29eddfe6865eafb8e (127.18.222.1:45627), 20d7112efcfa495c9baa82bd089f4928 (127.18.222.2:42835) I0918 20:39:09.147269 25012 tablet_bootstrap.cc:590] T 4b86906ac1084018a23f202664ca0276 P 20d7112efcfa495c9baa82bd089f4928: No blocks or log segments found. Creating new log. I0918 20:39:09.147366 25014 raft_consensus.cc:667] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [term 1 LEADER]: Becoming Leader. State: Replica: 4f96668e49f44cfe9cdfe6462347a594, State: Running, Role: LEADER I0918 20:39:09.148877 25036 raft_consensus.cc:2644] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:09.149821 25036 raft_consensus.cc:667] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 [term 1 LEADER]: Becoming Leader. State: Replica: 4f96668e49f44cfe9cdfe6462347a594, State: Running, Role: LEADER I0918 20:39:09.151306 25036 consensus_queue.cc:206] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } I0918 20:39:09.154038 25014 consensus_queue.cc:206] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } I0918 20:39:09.170821 24885 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "4b86906ac1084018a23f202664ca0276" candidate_uuid: "4f96668e49f44cfe9cdfe6462347a594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d7112efcfa495c9baa82bd089f4928" W0918 20:39:09.173758 24928 leader_election.cc:348] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 20d7112efcfa495c9baa82bd089f4928 (127.18.222.2:42835): Illegal state: must be running to vote when last-logged opid is not known I0918 20:39:09.185580 24886 raft_consensus.cc:2350] T 98003d0b22fb4a248daa8ba50ca38265 P 20d7112efcfa495c9baa82bd089f4928 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f96668e49f44cfe9cdfe6462347a594 in term 1. I0918 20:39:09.186854 24608 catalog_manager.cc:4131] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 reported cstate change: term changed from 0 to 1, leader changed from to 4f96668e49f44cfe9cdfe6462347a594 (127.18.222.3). New cstate: current_term: 1 leader_uuid: "4f96668e49f44cfe9cdfe6462347a594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } health_report { overall_health: HEALTHY } } } I0918 20:39:09.203799 25012 tablet_bootstrap.cc:439] T 4b86906ac1084018a23f202664ca0276 P 20d7112efcfa495c9baa82bd089f4928: No bootstrap required, opened a new log I0918 20:39:09.204309 25012 ts_tablet_manager.cc:1085] T 4b86906ac1084018a23f202664ca0276 P 20d7112efcfa495c9baa82bd089f4928: Time spent bootstrapping tablet: real 0.162s user 0.005s sys 0.004s I0918 20:39:09.206199 25012 raft_consensus.cc:340] T 4b86906ac1084018a23f202664ca0276 P 20d7112efcfa495c9baa82bd089f4928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } I0918 20:39:09.206907 25012 raft_consensus.cc:366] T 4b86906ac1084018a23f202664ca0276 P 20d7112efcfa495c9baa82bd089f4928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:09.207135 25012 raft_consensus.cc:705] T 4b86906ac1084018a23f202664ca0276 P 20d7112efcfa495c9baa82bd089f4928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d7112efcfa495c9baa82bd089f4928, State: Initialized, Role: FOLLOWER I0918 20:39:09.207864 25012 consensus_queue.cc:229] T 4b86906ac1084018a23f202664ca0276 P 20d7112efcfa495c9baa82bd089f4928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } I0918 20:39:09.215102 25012 ts_tablet_manager.cc:1113] T 4b86906ac1084018a23f202664ca0276 P 20d7112efcfa495c9baa82bd089f4928: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s I0918 20:39:09.242290 24608 catalog_manager.cc:4131] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 reported cstate change: term changed from 0 to 1, leader changed from to 4f96668e49f44cfe9cdfe6462347a594 (127.18.222.3). New cstate: current_term: 1 leader_uuid: "4f96668e49f44cfe9cdfe6462347a594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } health_report { overall_health: UNKNOWN } } } I0918 20:39:09.339431 19320 master.cc:247] Master@127.18.222.62:37057 shutting down... I0918 20:39:09.349733 24592 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.351325 24595 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.358624 24593 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.364006 24592 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.367038 24595 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.371467 24593 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.379287 24592 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.380529 24595 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.386307 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:09.386773 24593 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.387324 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:09.387796 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e: stopping tablet replica I0918 20:39:09.392427 24592 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.393115 24595 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.426054 19320 master.cc:258] Master@127.18.222.62:37057 shutdown complete. I0918 20:39:09.462265 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 152092us I0918 20:39:09.462653 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:09.462813 19320 fs_manager.cc:266] Using existing metadata directory in first data directory I0918 20:39:09.466027 19320 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s I0918 20:39:09.468111 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:09.484226 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0918 20:39:09.484534 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-0-root uuid: "c099560cc34545d88ee0f5408bf0078e" format_stamp: "Formatted at 2019-09-18 20:39:07 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:09.484872 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:09.507917 25020 raft_consensus.cc:472] T 4b86906ac1084018a23f202664ca0276 P 20d7112efcfa495c9baa82bd089f4928 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:09.508415 25020 raft_consensus.cc:494] T 4b86906ac1084018a23f202664ca0276 P 20d7112efcfa495c9baa82bd089f4928 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 } } peers { permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 } } peers { permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 } } I0918 20:39:09.510107 25020 leader_election.cc:296] T 4b86906ac1084018a23f202664ca0276 P 20d7112efcfa495c9baa82bd089f4928 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6071a1b02054de29eddfe6865eafb8e (127.18.222.1:45627), 4f96668e49f44cfe9cdfe6462347a594 (127.18.222.3:45255) I0918 20:39:09.526278 25017 consensus_queue.cc:984] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:09.548848 24812 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "4b86906ac1084018a23f202664ca0276" candidate_uuid: "20d7112efcfa495c9baa82bd089f4928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6071a1b02054de29eddfe6865eafb8e" is_pre_election: true I0918 20:39:09.549652 24812 raft_consensus.cc:2280] T 4b86906ac1084018a23f202664ca0276 P c6071a1b02054de29eddfe6865eafb8e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20d7112efcfa495c9baa82bd089f4928 in current term 1: Already voted for candidate 4f96668e49f44cfe9cdfe6462347a594 in this term. I0918 20:39:09.550951 25036 consensus_queue.cc:984] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:09.555492 24964 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "4b86906ac1084018a23f202664ca0276" candidate_uuid: "20d7112efcfa495c9baa82bd089f4928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f96668e49f44cfe9cdfe6462347a594" is_pre_election: true I0918 20:39:09.556394 24964 raft_consensus.cc:2311] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 20d7112efcfa495c9baa82bd089f4928 for term 1 because replica is either leader or believes a valid leader to be alive. I0918 20:39:09.557575 24846 leader_election.cc:310] T 4b86906ac1084018a23f202664ca0276 P 20d7112efcfa495c9baa82bd089f4928 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20d7112efcfa495c9baa82bd089f4928; no voters: 4f96668e49f44cfe9cdfe6462347a594, c6071a1b02054de29eddfe6865eafb8e I0918 20:39:09.558823 25020 raft_consensus.cc:2890] T 4b86906ac1084018a23f202664ca0276 P 20d7112efcfa495c9baa82bd089f4928 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:09.565555 25020 raft_consensus.cc:2592] T 4b86906ac1084018a23f202664ca0276 P 20d7112efcfa495c9baa82bd089f4928 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0918 20:39:09.590344 25037 consensus_queue.cc:984] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:39:09.604841 25032 consensus_queue.cc:984] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:09.608918 25037 consensus_queue.cc:984] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:09.669836 25036 consensus_queue.cc:984] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:09.833362 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:37057 I0918 20:39:09.836544 19320 webserver.cc:374] Webserver started at http://127.18.222.62:44375/ using document root and password file I0918 20:39:09.844080 19320 master.cc:247] Master@127.18.222.61:42813 shutting down... I0918 20:39:09.851395 24652 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.858482 24649 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.860188 24652 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.864153 25124 sys_catalog.cc:188] Verifying existing consensus state I0918 20:39:09.877043 24649 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.878051 24652 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.878857 25124 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e: Bootstrap starting. I0918 20:39:09.879365 24652 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.890909 24649 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.896705 24652 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.906625 24992 heartbeater.cc:322] Connected to a master server at 127.18.222.62:37057 I0918 20:39:09.908673 24649 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.910344 24652 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.915763 24652 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.918606 24837 heartbeater.cc:322] Connected to a master server at 127.18.222.62:37057 I0918 20:39:09.921726 24913 heartbeater.cc:322] Connected to a master server at 127.18.222.62:37057 I0918 20:39:09.922652 24649 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.930904 24652 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.931546 24652 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.935437 24649 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.942574 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:09.943224 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:09.943557 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc: stopping tablet replica I0918 20:39:09.948104 24649 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.946105 24652 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.952085 24652 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.952316 25124 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=6 ignored=0} orphaned_commits=0. Pending: 0 replicates I0918 20:39:09.956032 25124 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e: Bootstrap complete. I0918 20:39:09.958691 25124 raft_consensus.cc:340] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:09.960052 25124 raft_consensus.cc:705] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c099560cc34545d88ee0f5408bf0078e, State: Initialized, Role: FOLLOWER I0918 20:39:09.960991 25124 consensus_queue.cc:229] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:09.962790 24652 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:09.966771 25124 sys_catalog.cc:340] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:09.968645 25124 sys_catalog.cc:343] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:09.969887 25124 sys_catalog.cc:428] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:09.974157 25128 sys_catalog.cc:340] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:09.974952 25128 sys_catalog.cc:343] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:09.976392 25090 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" instance_seqno: 1568839148219987) as {username='slave'} at 127.0.0.1:39566; Asking this server to re-register. I0918 20:39:09.980648 25089 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" instance_seqno: 1568839147802738) as {username='slave'} at 127.0.0.1:39570; Asking this server to re-register. I0918 20:39:09.983184 24992 heartbeater.cc:406] Registering TS with master... I0918 20:39:09.984407 24837 heartbeater.cc:406] Registering TS with master... I0918 20:39:09.989733 25090 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" instance_seqno: 1568839147984755) as {username='slave'} at 127.0.0.1:39568; Asking this server to re-register. I0918 20:39:09.991495 25129 catalog_manager.cc:1153] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e: acquiring CA information for follower catalog manager: success I0918 20:39:10.033354 24913 heartbeater.cc:406] Registering TS with master... I0918 20:39:10.154815 25129 catalog_manager.cc:1181] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:10.161914 25089 ts_manager.cc:186] Registered new tserver with Master: 4f96668e49f44cfe9cdfe6462347a594 (127.18.222.3:45255) I0918 20:39:10.507479 19320 master.cc:258] Master@127.18.222.61:42813 shutdown complete. I0918 20:39:10.509472 25090 ts_manager.cc:186] Registered new tserver with Master: c6071a1b02054de29eddfe6865eafb8e (127.18.222.1:45627) I0918 20:39:10.524190 25088 ts_manager.cc:186] Registered new tserver with Master: 20d7112efcfa495c9baa82bd089f4928 (127.18.222.2:42835) W0918 20:39:10.634361 24845 outbound_call.cc:287] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.18.222.62:37057, user_credentials={real_user=slave}} blocked reactor thread for 77774.6us I0918 20:39:10.851191 25137 raft_consensus.cc:472] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c099560cc34545d88ee0f5408bf0078e) I0918 20:39:10.852061 25137 raft_consensus.cc:494] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:10.854069 25137 leader_election.cc:296] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c099560cc34545d88ee0f5408bf0078e (127.18.222.62:37057), b6ca09a1f9e5496f95857e93a1bfddfc (127.18.222.61:42813) W0918 20:39:10.882171 24709 leader_election.cc:341] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b6ca09a1f9e5496f95857e93a1bfddfc (127.18.222.61:42813): Network error: Client connection negotiation failed: client connection to 127.18.222.61:42813: connect: Connection refused (error 111) I0918 20:39:10.900194 25100 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b0782873fac45c19186dd7e9966b568" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "c099560cc34545d88ee0f5408bf0078e" is_pre_election: true I0918 20:39:10.901405 25100 raft_consensus.cc:2350] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b0782873fac45c19186dd7e9966b568 in term 1. I0918 20:39:10.902909 24707 leader_election.cc:310] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b0782873fac45c19186dd7e9966b568, c099560cc34545d88ee0f5408bf0078e; no voters: b6ca09a1f9e5496f95857e93a1bfddfc I0918 20:39:10.903970 25137 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0918 20:39:10.904266 25137 raft_consensus.cc:472] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c099560cc34545d88ee0f5408bf0078e) I0918 20:39:10.905082 25137 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 1 FOLLOWER]: Advancing to term 2 I0918 20:39:10.914005 25137 raft_consensus.cc:494] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:10.916319 25100 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b0782873fac45c19186dd7e9966b568" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "c099560cc34545d88ee0f5408bf0078e" I0918 20:39:10.916939 25100 raft_consensus.cc:2890] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 1 FOLLOWER]: Advancing to term 2 W0918 20:39:10.917116 24709 leader_election.cc:341] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b6ca09a1f9e5496f95857e93a1bfddfc (127.18.222.61:42813): Network error: Client connection negotiation failed: client connection to 127.18.222.61:42813: connect: Connection refused (error 111) I0918 20:39:10.919801 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 152592us I0918 20:39:10.920486 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:10.920733 19320 fs_manager.cc:266] Using existing metadata directory in first data directory I0918 20:39:10.923816 25137 leader_election.cc:296] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [CANDIDATE]: Term 2 election: Requested vote from peers c099560cc34545d88ee0f5408bf0078e (127.18.222.62:37057), b6ca09a1f9e5496f95857e93a1bfddfc (127.18.222.61:42813) I0918 20:39:10.924387 25100 raft_consensus.cc:2350] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0782873fac45c19186dd7e9966b568 in term 2. I0918 20:39:10.925659 24707 leader_election.cc:310] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b0782873fac45c19186dd7e9966b568, c099560cc34545d88ee0f5408bf0078e; no voters: b6ca09a1f9e5496f95857e93a1bfddfc I0918 20:39:10.926371 25137 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 2 FOLLOWER]: Leader election won for term 2 I0918 20:39:10.935741 25137 raft_consensus.cc:667] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 2 LEADER]: Becoming Leader. State: Replica: 1b0782873fac45c19186dd7e9966b568, State: Running, Role: LEADER I0918 20:39:10.940150 19320 fs_manager.cc:399] Time spent opening directory manager: real 0.018s user 0.009s sys 0.009s I0918 20:39:10.939878 25137 consensus_queue.cc:206] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:10.942526 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:10.951423 25142 sys_catalog.cc:340] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b0782873fac45c19186dd7e9966b568. Latest consensus state: current_term: 2 leader_uuid: "1b0782873fac45c19186dd7e9966b568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:10.953212 25142 sys_catalog.cc:343] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [sys.catalog]: This master's current role is: LEADER I0918 20:39:10.961161 25144 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:39:10.961264 19320 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s I0918 20:39:10.963904 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-1-root uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" format_stamp: "Formatted at 2019-09-18 20:39:07 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:10.968518 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:10.971801 25144 catalog_manager.cc:387] Loaded metadata for table ksck-test-table [id=7aa24a7d75d54b7889f05c617afd1dfb] I0918 20:39:10.977654 25144 catalog_manager.cc:441] Loaded metadata for tablet 4b86906ac1084018a23f202664ca0276 (table ksck-test-table [id=7aa24a7d75d54b7889f05c617afd1dfb]) I0918 20:39:10.979183 25144 catalog_manager.cc:441] Loaded metadata for tablet 8119ec3019aa43a78aace2e7ed85055d (table ksck-test-table [id=7aa24a7d75d54b7889f05c617afd1dfb]) I0918 20:39:10.980589 25144 catalog_manager.cc:441] Loaded metadata for tablet 98003d0b22fb4a248daa8ba50ca38265 (table ksck-test-table [id=7aa24a7d75d54b7889f05c617afd1dfb]) I0918 20:39:10.981624 25144 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:39:10.986884 25144 catalog_manager.cc:1106] Loading token signing keys... I0918 20:39:10.990537 25144 catalog_manager.cc:4457] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568: Loaded TSK: 0 I0918 20:39:10.992189 25144 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:39:11.009876 24835 heartbeater.cc:447] Master 127.18.222.60:45381 was elected leader, sending a full tablet report... I0918 20:39:11.009980 24988 heartbeater.cc:447] Master 127.18.222.60:45381 was elected leader, sending a full tablet report... I0918 20:39:11.019354 24909 heartbeater.cc:447] Master 127.18.222.60:45381 was elected leader, sending a full tablet report... I0918 20:39:11.133703 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:42813 I0918 20:39:11.136621 19320 webserver.cc:374] Webserver started at http://127.18.222.61:44145/ using document root and password file I0918 20:39:11.138949 19320 master.cc:247] Master@127.18.222.60:45381 shutting down... I0918 20:39:11.140868 24709 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.147055 24707 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.147964 24707 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.163331 24709 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.239907 24709 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.248628 24707 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.277972 24709 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.288733 24707 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.294212 24709 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.301919 24707 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.308194 24707 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.319301 24707 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.319806 24709 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.330965 24707 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.342425 24707 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.346995 24709 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.350817 25198 sys_catalog.cc:188] Verifying existing consensus state I0918 20:39:11.370007 24707 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.395665 24707 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.411041 25198 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc: Bootstrap starting. I0918 20:39:11.416296 24709 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.416819 24707 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.419591 24707 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.426159 24989 heartbeater.cc:322] Connected to a master server at 127.18.222.61:42813 I0918 20:39:11.440172 24707 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.440968 24836 heartbeater.cc:322] Connected to a master server at 127.18.222.61:42813 I0918 20:39:11.445190 24910 heartbeater.cc:322] Connected to a master server at 127.18.222.61:42813 I0918 20:39:11.445241 24709 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.451400 24707 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.457044 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 2 LEADER]: Raft consensus shutting down. I0918 20:39:11.457999 19320 pending_rounds.cc:64] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568: Trying to abort 1 pending transactions. I0918 20:39:11.458386 19320 pending_rounds.cc:70] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568: Aborting transaction as it isn't in flight: id { term: 2 index: 9 } timestamp: 6425965162258554880 op_type: NO_OP noop_request { } I0918 20:39:11.458997 19320 raft_consensus.cc:2727] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 2 LEADER]: NO_OP replication failed: Aborted: Transaction aborted I0918 20:39:11.459275 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 2 FOLLOWER]: Raft consensus is shut down! I0918 20:39:11.462435 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568: stopping tablet replica I0918 20:39:11.465662 24707 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.466580 24707 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.472656 24709 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.479969 24836 heartbeater.cc:322] Connected to a master server at 127.18.222.61:42813 I0918 20:39:11.485718 24709 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0918 20:39:11.510679 25198 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=6 ignored=0} orphaned_commits=0. Pending: 0 replicates I0918 20:39:11.511256 19320 master.cc:258] Master@127.18.222.60:45381 shutdown complete. I0918 20:39:11.512080 25198 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc: Bootstrap complete. I0918 20:39:11.519017 25198 raft_consensus.cc:340] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:11.519981 25198 raft_consensus.cc:705] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ca09a1f9e5496f95857e93a1bfddfc, State: Initialized, Role: FOLLOWER I0918 20:39:11.520824 25198 consensus_queue.cc:229] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:11.524704 25198 sys_catalog.cc:340] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:11.525010 25205 sys_catalog.cc:340] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:11.525554 25198 sys_catalog.cc:343] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:11.525818 25205 sys_catalog.cc:343] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:11.526945 25198 sys_catalog.cc:428] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:11.527216 25175 raft_consensus.cc:2890] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 1 FOLLOWER]: Advancing to term 2 I0918 20:39:11.536729 24989 heartbeater.cc:322] Connected to a master server at 127.18.222.61:42813 I0918 20:39:11.543618 25175 raft_consensus.cc:1184] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 2 FOLLOWER]: Refusing update from remote peer 1b0782873fac45c19186dd7e9966b568: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) W0918 20:39:11.548647 25146 connection.cc:562] Connection torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41582 (request call id 1) could send its response I0918 20:39:11.549343 25165 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" instance_seqno: 1568839147984755) as {username='slave'} at 127.0.0.1:41580; Asking this server to re-register. I0918 20:39:11.550839 24910 heartbeater.cc:406] Registering TS with master... I0918 20:39:11.551460 25164 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" instance_seqno: 1568839147802738) as {username='slave'} at 127.0.0.1:41578; Asking this server to re-register. I0918 20:39:11.552366 25163 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" instance_seqno: 1568839148219987) as {username='slave'} at 127.0.0.1:41576; Asking this server to re-register. I0918 20:39:11.553414 24836 heartbeater.cc:406] Registering TS with master... I0918 20:39:11.662595 25164 ts_manager.cc:186] Registered new tserver with Master: 20d7112efcfa495c9baa82bd089f4928 (127.18.222.2:42835) I0918 20:39:11.672933 24989 heartbeater.cc:406] Registering TS with master... I0918 20:39:11.755362 25206 catalog_manager.cc:1153] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc: acquiring CA information for follower catalog manager: success I0918 20:39:11.777015 25163 ts_manager.cc:186] Registered new tserver with Master: c6071a1b02054de29eddfe6865eafb8e (127.18.222.1:45627) I0918 20:39:11.800004 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 153092us I0918 20:39:11.801079 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:11.801681 19320 fs_manager.cc:266] Using existing metadata directory in first data directory I0918 20:39:11.803998 25206 catalog_manager.cc:1181] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:11.817816 19320 fs_manager.cc:399] Time spent opening directory manager: real 0.008s user 0.000s sys 0.002s I0918 20:39:11.818403 25165 ts_manager.cc:186] Registered new tserver with Master: 4f96668e49f44cfe9cdfe6462347a594 (127.18.222.3:45255) I0918 20:39:11.820046 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:11.835029 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:39:11.835640 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-2-root uuid: "1b0782873fac45c19186dd7e9966b568" format_stamp: "Formatted at 2019-09-18 20:39:07 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:11.836133 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:12.024197 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:45381 I0918 20:39:12.034191 19320 webserver.cc:374] Webserver started at http://127.18.222.60:39679/ using document root and password file I0918 20:39:12.076143 25268 sys_catalog.cc:188] Verifying existing consensus state I0918 20:39:12.090608 24988 heartbeater.cc:322] Connected to a master server at 127.18.222.60:45381 I0918 20:39:12.092787 24909 heartbeater.cc:322] Connected to a master server at 127.18.222.60:45381 I0918 20:39:12.095232 24835 heartbeater.cc:322] Connected to a master server at 127.18.222.60:45381 I0918 20:39:12.103837 25268 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568: Bootstrap starting. I0918 20:39:12.111546 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 153592us I0918 20:39:12.116293 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:12.116605 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:12.117597 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:12.118194 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:12.122496 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/ts-3-root/instance: uuid: "2ffe169c7ea24f32b0ba24da590e6d2b" format_stamp: "Formatted at 2019-09-18 20:39:12 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:12.143144 25268 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=6 ignored=0} orphaned_commits=0. Pending: 1 replicates I0918 20:39:12.144454 25268 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568: Bootstrap complete. I0918 20:39:12.144901 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.022s user 0.012s sys 0.003s I0918 20:39:12.147471 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:12.147433 25268 raft_consensus.cc:340] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:12.148672 25268 raft_consensus.cc:705] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0782873fac45c19186dd7e9966b568, State: Initialized, Role: FOLLOWER I0918 20:39:12.149425 25268 consensus_queue.cc:229] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:12.155239 25268 sys_catalog.cc:340] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:12.156184 25268 sys_catalog.cc:343] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:12.157476 25268 sys_catalog.cc:428] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:12.155457 25271 sys_catalog.cc:340] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:12.166247 25271 sys_catalog.cc:343] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:12.166927 25231 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "c6071a1b02054de29eddfe6865eafb8e" instance_seqno: 1568839147802738) as {username='slave'} at 127.0.0.1:34354; Asking this server to re-register. I0918 20:39:12.168555 24835 heartbeater.cc:406] Registering TS with master... I0918 20:39:12.172597 19320 fs_manager.cc:419] Time spent opening block manager: real 0.008s user 0.003s sys 0.001s I0918 20:39:12.258985 25232 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "20d7112efcfa495c9baa82bd089f4928" instance_seqno: 1568839147984755) as {username='slave'} at 127.0.0.1:34356; Asking this server to re-register. I0918 20:39:12.270995 24909 heartbeater.cc:406] Registering TS with master... I0918 20:39:12.278143 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/ts-3-root uuid: "2ffe169c7ea24f32b0ba24da590e6d2b" format_stamp: "Formatted at 2019-09-18 20:39:12 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:12.288103 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/ts-3-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/ts-3-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1568839089958813-19320/minicluster-data/ts-3-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:12.376547 25231 ts_manager.cc:186] Registered new tserver with Master: c6071a1b02054de29eddfe6865eafb8e (127.18.222.1:45627) W0918 20:39:12.436499 24928 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.222.1:45627, user_credentials={real_user=slave}} blocked reactor thread for 175890us I0918 20:39:12.437053 25231 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "4f96668e49f44cfe9cdfe6462347a594" instance_seqno: 1568839148219987) as {username='slave'} at 127.0.0.1:34352; Asking this server to re-register. I0918 20:39:12.443364 24988 heartbeater.cc:406] Registering TS with master... I0918 20:39:12.525386 25272 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568: acquiring CA information for follower catalog manager: success I0918 20:39:12.533717 25232 ts_manager.cc:186] Registered new tserver with Master: 20d7112efcfa495c9baa82bd089f4928 (127.18.222.2:42835) W0918 20:39:12.542786 24928 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.222.1:45627, user_credentials={real_user=slave}} blocked reactor thread for 92441.5us I0918 20:39:12.571393 25230 ts_manager.cc:186] Registered new tserver with Master: 4f96668e49f44cfe9cdfe6462347a594 (127.18.222.3:45255) I0918 20:39:12.580153 25272 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:12.653128 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:12.653584 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:12.979074 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.4:32911 I0918 20:39:12.984743 19320 webserver.cc:374] Webserver started at http://127.18.222.4:40131/ using document root and password file I0918 20:39:13.237694 25342 heartbeater.cc:322] Connected to a master server at 127.18.222.60:45381 I0918 20:39:13.245698 25342 heartbeater.cc:406] Registering TS with master... I0918 20:39:13.239389 25343 heartbeater.cc:322] Connected to a master server at 127.18.222.61:42813 I0918 20:39:13.247790 25343 heartbeater.cc:406] Registering TS with master... I0918 20:39:13.248494 25343 heartbeater.cc:455] Master 127.18.222.61:42813 requested a full tablet report, sending... I0918 20:39:13.253204 25342 heartbeater.cc:455] Master 127.18.222.60:45381 requested a full tablet report, sending... I0918 20:39:13.304891 25352 raft_consensus.cc:472] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:13.328586 25352 raft_consensus.cc:494] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:13.365751 25354 raft_consensus.cc:472] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:13.367102 25354 raft_consensus.cc:494] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:13.371997 25354 leader_election.cc:296] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c099560cc34545d88ee0f5408bf0078e (127.18.222.62:37057), 1b0782873fac45c19186dd7e9966b568 (127.18.222.60:45381) I0918 20:39:13.398964 25352 leader_election.cc:296] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b6ca09a1f9e5496f95857e93a1bfddfc (127.18.222.61:42813), 1b0782873fac45c19186dd7e9966b568 (127.18.222.60:45381) I0918 20:39:13.412992 25163 ts_manager.cc:186] Registered new tserver with Master: 2ffe169c7ea24f32b0ba24da590e6d2b (127.18.222.4:32911) I0918 20:39:13.576860 25230 ts_manager.cc:186] Registered new tserver with Master: 2ffe169c7ea24f32b0ba24da590e6d2b (127.18.222.4:32911) I0918 20:39:13.589218 25348 heartbeater.cc:322] Connected to a master server at 127.18.222.62:37057 I0918 20:39:13.589884 25348 heartbeater.cc:406] Registering TS with master... I0918 20:39:13.590672 25348 heartbeater.cc:455] Master 127.18.222.62:37057 requested a full tablet report, sending... I0918 20:39:13.586588 25365 raft_consensus.cc:472] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:13.592129 25365 raft_consensus.cc:494] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:13.594945 25365 leader_election.cc:296] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c099560cc34545d88ee0f5408bf0078e (127.18.222.62:37057), b6ca09a1f9e5496f95857e93a1bfddfc (127.18.222.61:42813) I0918 20:39:13.928268 25100 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "c099560cc34545d88ee0f5408bf0078e" is_pre_election: true I0918 20:39:13.929293 25100 raft_consensus.cc:2350] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6ca09a1f9e5496f95857e93a1bfddfc in term 2. I0918 20:39:13.931239 25090 ts_manager.cc:186] Registered new tserver with Master: 2ffe169c7ea24f32b0ba24da590e6d2b (127.18.222.4:32911) I0918 20:39:13.960542 25147 leader_election.cc:310] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6ca09a1f9e5496f95857e93a1bfddfc, c099560cc34545d88ee0f5408bf0078e; no voters: I0918 20:39:13.995347 25242 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c099560cc34545d88ee0f5408bf0078e" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1b0782873fac45c19186dd7e9966b568" is_pre_election: true I0918 20:39:13.996754 25242 raft_consensus.cc:2297] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c099560cc34545d88ee0f5408bf0078e for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8. W0918 20:39:14.174752 25146 thread.cc:599] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.597s user 0.005s sys 0.022s W0918 20:39:14.175427 25146 thread.cc:566] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.630s user 0.005s sys 0.022s W0918 20:39:14.173065 25214 thread.cc:599] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.006s sys 0.025s W0918 20:39:14.189424 25149 thread.cc:599] server-negotiator [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.004s sys 0.021s W0918 20:39:14.228915 25214 thread.cc:566] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.632s user 0.010s sys 0.036s W0918 20:39:14.229259 25149 thread.cc:566] server-negotiator [worker] (thread pool) Time spent starting thread: real 0.631s user 0.004s sys 0.028s W0918 20:39:14.164899 25147 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.222.62:37057, user_credentials={real_user=slave}} blocked reactor thread for 204662us I0918 20:39:14.191982 25377 raft_consensus.cc:2644] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 2 FOLLOWER]: Leader pre-election won for term 3 I0918 20:39:14.246712 25377 raft_consensus.cc:472] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:14.247143 25377 raft_consensus.cc:2890] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 2 FOLLOWER]: Advancing to term 3 W0918 20:39:14.182274 25216 thread.cc:599] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.023s sys 0.062s W0918 20:39:14.251075 25216 thread.cc:566] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.655s user 0.023s sys 0.068s I0918 20:39:14.260834 25100 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b0782873fac45c19186dd7e9966b568" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "c099560cc34545d88ee0f5408bf0078e" is_pre_election: true I0918 20:39:14.261643 25100 raft_consensus.cc:2350] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b0782873fac45c19186dd7e9966b568 in term 2. I0918 20:39:14.263375 25242 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1b0782873fac45c19186dd7e9966b568" is_pre_election: true I0918 20:39:14.264384 25242 raft_consensus.cc:2297] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b6ca09a1f9e5496f95857e93a1bfddfc for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8. I0918 20:39:14.274825 25377 raft_consensus.cc:494] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:14.293792 25175 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b0782873fac45c19186dd7e9966b568" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" is_pre_election: true I0918 20:39:14.294648 25175 raft_consensus.cc:2280] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b0782873fac45c19186dd7e9966b568 in current term 3: Already voted for candidate b6ca09a1f9e5496f95857e93a1bfddfc in this term. I0918 20:39:14.279073 25214 leader_election.cc:310] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0782873fac45c19186dd7e9966b568, c099560cc34545d88ee0f5408bf0078e; no voters: I0918 20:39:14.297422 25175 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c099560cc34545d88ee0f5408bf0078e" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" is_pre_election: true I0918 20:39:14.298207 25175 raft_consensus.cc:2280] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c099560cc34545d88ee0f5408bf0078e in current term 3: Already voted for candidate b6ca09a1f9e5496f95857e93a1bfddfc in this term. I0918 20:39:14.301329 25059 leader_election.cc:310] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c099560cc34545d88ee0f5408bf0078e; no voters: 1b0782873fac45c19186dd7e9966b568, b6ca09a1f9e5496f95857e93a1bfddfc I0918 20:39:14.314787 25100 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "c099560cc34545d88ee0f5408bf0078e" I0918 20:39:14.315431 25100 raft_consensus.cc:2890] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 2 FOLLOWER]: Advancing to term 3 I0918 20:39:14.324371 25100 raft_consensus.cc:2350] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6ca09a1f9e5496f95857e93a1bfddfc in term 3. I0918 20:39:14.331655 25147 leader_election.cc:310] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6ca09a1f9e5496f95857e93a1bfddfc, c099560cc34545d88ee0f5408bf0078e; no voters: I0918 20:39:14.341869 25377 leader_election.cc:296] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [CANDIDATE]: Term 3 election: Requested vote from peers c099560cc34545d88ee0f5408bf0078e (127.18.222.62:37057), 1b0782873fac45c19186dd7e9966b568 (127.18.222.60:45381) I0918 20:39:14.342531 25377 raft_consensus.cc:2644] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 3 FOLLOWER]: Leader election won for term 3 I0918 20:39:14.342963 25377 raft_consensus.cc:667] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 3 LEADER]: Becoming Leader. State: Replica: b6ca09a1f9e5496f95857e93a1bfddfc, State: Running, Role: LEADER I0918 20:39:14.342941 25242 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1b0782873fac45c19186dd7e9966b568" I0918 20:39:14.343984 25242 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 2 FOLLOWER]: Advancing to term 3 I0918 20:39:14.345525 25377 consensus_queue.cc:206] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } I0918 20:39:14.355418 25242 raft_consensus.cc:2297] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b6ca09a1f9e5496f95857e93a1bfddfc for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8. W0918 20:39:14.360741 25214 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.222.62:37057, user_credentials={real_user=slave}} blocked reactor thread for 81761.9us I0918 20:39:14.364663 25379 raft_consensus.cc:2607] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won I0918 20:39:14.390111 25380 raft_consensus.cc:2592] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority W0918 20:39:14.408036 25147 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.222.62:37057, user_credentials={real_user=slave}} blocked reactor thread for 76477.9us I0918 20:39:14.410622 25381 sys_catalog.cc:340] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6ca09a1f9e5496f95857e93a1bfddfc. Latest consensus state: current_term: 3 leader_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:14.412336 25381 sys_catalog.cc:343] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [sys.catalog]: This master's current role is: LEADER I0918 20:39:14.469682 25383 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:39:14.486524 25383 catalog_manager.cc:387] Loaded metadata for table ksck-test-table [id=7aa24a7d75d54b7889f05c617afd1dfb] I0918 20:39:14.490794 25383 catalog_manager.cc:441] Loaded metadata for tablet 4b86906ac1084018a23f202664ca0276 (table ksck-test-table [id=7aa24a7d75d54b7889f05c617afd1dfb]) I0918 20:39:14.533339 25383 catalog_manager.cc:441] Loaded metadata for tablet 8119ec3019aa43a78aace2e7ed85055d (table ksck-test-table [id=7aa24a7d75d54b7889f05c617afd1dfb]) I0918 20:39:14.542147 25383 catalog_manager.cc:441] Loaded metadata for tablet 98003d0b22fb4a248daa8ba50ca38265 (table ksck-test-table [id=7aa24a7d75d54b7889f05c617afd1dfb]) I0918 20:39:14.543231 25383 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:39:14.552954 25383 catalog_manager.cc:1106] Loading token signing keys... I0918 20:39:14.556138 25383 catalog_manager.cc:4457] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc: Loaded TSK: 0 I0918 20:39:14.560678 25383 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:39:14.569099 25164 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41750 I0918 20:39:14.576813 24836 heartbeater.cc:447] Master 127.18.222.61:42813 was elected leader, sending a full tablet report... I0918 20:39:14.583838 25343 heartbeater.cc:447] Master 127.18.222.61:42813 was elected leader, sending a full tablet report... I0918 20:39:14.588388 24910 heartbeater.cc:447] Master 127.18.222.61:42813 was elected leader, sending a full tablet report... I0918 20:39:14.589035 24989 heartbeater.cc:447] Master 127.18.222.61:42813 was elected leader, sending a full tablet report... I0918 20:39:14.778324 25100 raft_consensus.cc:1184] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 3 FOLLOWER]: Refusing update from remote peer b6ca09a1f9e5496f95857e93a1bfddfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch) I0918 20:39:14.780242 25381 consensus_queue.cc:984] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I0918 20:39:14.813604 25380 sys_catalog.cc:340] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6ca09a1f9e5496f95857e93a1bfddfc. Latest consensus state: current_term: 3 leader_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:14.814390 25380 sys_catalog.cc:343] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:14.845448 25381 sys_catalog.cc:340] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:14.846293 25381 sys_catalog.cc:343] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [sys.catalog]: This master's current role is: LEADER I0918 20:39:14.848723 25242 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 3 FOLLOWER]: Refusing update from remote peer b6ca09a1f9e5496f95857e93a1bfddfc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 9. (term mismatch) I0918 20:39:14.849192 25242 pending_rounds.cc:78] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568: Aborting all transactions after (but not including) 8 I0918 20:39:14.849416 25242 pending_rounds.cc:99] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568: Aborting uncommitted NO_OP operation due to leader change: 2.9 I0918 20:39:14.849644 25242 raft_consensus.cc:2727] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader I0918 20:39:14.856920 25381 consensus_queue.cc:984] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I0918 20:39:14.880014 25380 sys_catalog.cc:340] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:14.880846 25380 sys_catalog.cc:343] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:14.889639 25394 sys_catalog.cc:340] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:14.890542 25394 sys_catalog.cc:343] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:14.890666 25379 sys_catalog.cc:340] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6ca09a1f9e5496f95857e93a1bfddfc. Latest consensus state: current_term: 3 leader_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:14.891649 25379 sys_catalog.cc:343] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:15.344684 25381 sys_catalog.cc:340] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c099560cc34545d88ee0f5408bf0078e" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37057 } } peers { permanent_uuid: "b6ca09a1f9e5496f95857e93a1bfddfc" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 42813 } } peers { permanent_uuid: "1b0782873fac45c19186dd7e9966b568" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45381 } } } I0918 20:39:15.345743 25381 sys_catalog.cc:343] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [sys.catalog]: This master's current role is: LEADER I0918 20:39:16.390832 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:16.465715 24773 messenger.cc:381] Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer W0918 20:39:16.466941 24928 consensus_peers.cc:458] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 -> Peer c6071a1b02054de29eddfe6865eafb8e (127.18.222.1:45627): Couldn't send request to peer c6071a1b02054de29eddfe6865eafb8e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry. I0918 20:39:16.574503 24773 messenger.cc:381] Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer W0918 20:39:16.575882 24928 consensus_peers.cc:458] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 -> Peer c6071a1b02054de29eddfe6865eafb8e (127.18.222.1:45627): Couldn't send request to peer c6071a1b02054de29eddfe6865eafb8e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry. I0918 20:39:16.798897 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:16.799311 19320 tablet_replica.cc:273] T 4b86906ac1084018a23f202664ca0276 P c6071a1b02054de29eddfe6865eafb8e: stopping tablet replica I0918 20:39:16.799851 19320 raft_consensus.cc:2147] T 4b86906ac1084018a23f202664ca0276 P c6071a1b02054de29eddfe6865eafb8e [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:16.800151 19320 raft_consensus.cc:2174] T 4b86906ac1084018a23f202664ca0276 P c6071a1b02054de29eddfe6865eafb8e [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:16.802942 19320 tablet_replica.cc:273] T 98003d0b22fb4a248daa8ba50ca38265 P c6071a1b02054de29eddfe6865eafb8e: stopping tablet replica I0918 20:39:16.803490 19320 raft_consensus.cc:2147] T 98003d0b22fb4a248daa8ba50ca38265 P c6071a1b02054de29eddfe6865eafb8e [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:16.803992 19320 raft_consensus.cc:2174] T 98003d0b22fb4a248daa8ba50ca38265 P c6071a1b02054de29eddfe6865eafb8e [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:16.807142 19320 tablet_replica.cc:273] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e: stopping tablet replica I0918 20:39:16.807687 19320 raft_consensus.cc:2147] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:16.808830 19320 raft_consensus.cc:2174] T 8119ec3019aa43a78aace2e7ed85055d P c6071a1b02054de29eddfe6865eafb8e [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:16.845974 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:16.846292 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:16.846845 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:16.870167 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:17.001287 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:17.001981 19320 tablet_replica.cc:273] T 4b86906ac1084018a23f202664ca0276 P 20d7112efcfa495c9baa82bd089f4928: stopping tablet replica I0918 20:39:17.002699 19320 raft_consensus.cc:2147] T 4b86906ac1084018a23f202664ca0276 P 20d7112efcfa495c9baa82bd089f4928 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:17.003230 19320 raft_consensus.cc:2174] T 4b86906ac1084018a23f202664ca0276 P 20d7112efcfa495c9baa82bd089f4928 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:17.006618 19320 tablet_replica.cc:273] T 98003d0b22fb4a248daa8ba50ca38265 P 20d7112efcfa495c9baa82bd089f4928: stopping tablet replica I0918 20:39:17.007133 19320 raft_consensus.cc:2147] T 98003d0b22fb4a248daa8ba50ca38265 P 20d7112efcfa495c9baa82bd089f4928 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:17.007483 19320 raft_consensus.cc:2174] T 98003d0b22fb4a248daa8ba50ca38265 P 20d7112efcfa495c9baa82bd089f4928 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:17.009766 19320 tablet_replica.cc:273] T 8119ec3019aa43a78aace2e7ed85055d P 20d7112efcfa495c9baa82bd089f4928: stopping tablet replica I0918 20:39:17.010186 19320 raft_consensus.cc:2147] T 8119ec3019aa43a78aace2e7ed85055d P 20d7112efcfa495c9baa82bd089f4928 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:17.010607 19320 raft_consensus.cc:2174] T 8119ec3019aa43a78aace2e7ed85055d P 20d7112efcfa495c9baa82bd089f4928 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:17.043509 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:17.043768 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:17.044139 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:17.061906 19320 tablet_server.cc:155] TabletServer shutting down... W0918 20:39:17.104079 24928 consensus_peers.cc:458] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 -> Peer 20d7112efcfa495c9baa82bd089f4928 (127.18.222.2:42835): Couldn't send request to peer 20d7112efcfa495c9baa82bd089f4928. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.2:42835: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0918 20:39:17.134999 24928 consensus_peers.cc:458] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 -> Peer 20d7112efcfa495c9baa82bd089f4928 (127.18.222.2:42835): Couldn't send request to peer 20d7112efcfa495c9baa82bd089f4928. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.2:42835: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:39:17.137511 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:17.137984 19320 tablet_replica.cc:273] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594: stopping tablet replica I0918 20:39:17.138403 19320 raft_consensus.cc:2147] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:17.139171 19320 raft_consensus.cc:2174] T 4b86906ac1084018a23f202664ca0276 P 4f96668e49f44cfe9cdfe6462347a594 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:17.142542 19320 tablet_replica.cc:273] T 8119ec3019aa43a78aace2e7ed85055d P 4f96668e49f44cfe9cdfe6462347a594: stopping tablet replica I0918 20:39:17.143059 19320 raft_consensus.cc:2147] T 8119ec3019aa43a78aace2e7ed85055d P 4f96668e49f44cfe9cdfe6462347a594 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:17.143438 19320 raft_consensus.cc:2174] T 8119ec3019aa43a78aace2e7ed85055d P 4f96668e49f44cfe9cdfe6462347a594 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:17.146288 19320 tablet_replica.cc:273] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594: stopping tablet replica I0918 20:39:17.146847 19320 raft_consensus.cc:2147] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:17.147648 19320 raft_consensus.cc:2174] T 98003d0b22fb4a248daa8ba50ca38265 P 4f96668e49f44cfe9cdfe6462347a594 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:17.163846 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:17.164072 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:17.164435 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:17.180838 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:17.220597 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:17.236577 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:17.236807 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:17.237234 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:17.244837 19320 master.cc:247] Master@127.18.222.62:37057 shutting down... I0918 20:39:17.270573 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 3 FOLLOWER]: Raft consensus shutting down. I0918 20:39:17.270987 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e [term 3 FOLLOWER]: Raft consensus is shut down! I0918 20:39:17.271270 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P c099560cc34545d88ee0f5408bf0078e: stopping tablet replica I0918 20:39:17.273804 25059 messenger.cc:381] Service unavailable: service kudu.consensus.ConsensusService not registered on Master W0918 20:39:17.274718 25147 consensus_peers.cc:458] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc -> Peer c099560cc34545d88ee0f5408bf0078e (127.18.222.62:37057): Couldn't send request to peer c099560cc34545d88ee0f5408bf0078e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry. I0918 20:39:17.290045 19320 master.cc:258] Master@127.18.222.62:37057 shutdown complete. I0918 20:39:17.301004 19320 master.cc:247] Master@127.18.222.61:42813 shutting down... I0918 20:39:17.327297 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 3 LEADER]: Raft consensus shutting down. I0918 20:39:17.328150 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc [term 3 FOLLOWER]: Raft consensus is shut down! I0918 20:39:17.328423 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P b6ca09a1f9e5496f95857e93a1bfddfc: stopping tablet replica I0918 20:39:17.347127 19320 master.cc:258] Master@127.18.222.61:42813 shutdown complete. I0918 20:39:17.359127 19320 master.cc:247] Master@127.18.222.60:45381 shutting down... I0918 20:39:17.383314 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 3 FOLLOWER]: Raft consensus shutting down. I0918 20:39:17.383891 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568 [term 3 FOLLOWER]: Raft consensus is shut down! I0918 20:39:17.384228 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 1b0782873fac45c19186dd7e9966b568: stopping tablet replica I0918 20:39:17.404029 19320 master.cc:258] Master@127.18.222.60:45381 shutdown complete. [ OK ] RemoteKsckTest.TestClusterWithLocation (10212 ms) [ RUN ] RemoteKsckTest.TestFilterOnNoTableCluster I0918 20:39:17.443310 19320 test_util.cc:212] Using random seed: -726635824 I0918 20:39:17.444142 19320 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.18.222.62:36069,127.18.222.61:38137,127.18.222.60:41641 I0918 20:39:17.458461 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 156092us I0918 20:39:17.458798 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:17.458936 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:17.459118 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:17.459233 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:17.460291 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/master-0-root/instance: uuid: "b8424e7d64a648f3952fd83c45ad7fc4" format_stamp: "Formatted at 2019-09-18 20:39:17 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:17.468819 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.005s sys 0.003s I0918 20:39:17.470345 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:17.477300 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:39:17.477476 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/master-0-root uuid: "b8424e7d64a648f3952fd83c45ad7fc4" format_stamp: "Formatted at 2019-09-18 20:39:17 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:17.477706 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:17.544111 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:36069 I0918 20:39:17.545898 19320 webserver.cc:374] Webserver started at http://127.18.222.62:43185/ using document root and password file I0918 20:39:17.547659 25489 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:17.553000 25489 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:17.564777 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 156092us I0918 20:39:17.565312 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:17.565536 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:17.565801 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:17.565973 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:17.567211 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/master-1-root/instance: uuid: "5f649e2399b743d9a53a42952539e941" format_stamp: "Formatted at 2019-09-18 20:39:17 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:17.574199 25489 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:17.576056 25489 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:38137: Network error: Client connection negotiation failed: client connection to 127.18.222.61:38137: connect: Connection refused (error 111) I0918 20:39:17.581192 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.014s user 0.010s sys 0.006s I0918 20:39:17.583849 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:17.596370 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0918 20:39:17.596575 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/master-1-root uuid: "5f649e2399b743d9a53a42952539e941" format_stamp: "Formatted at 2019-09-18 20:39:17 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:17.596880 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:17.631450 25489 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } attempt: 1 W0918 20:39:17.634016 25489 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:38137: Network error: Client connection negotiation failed: client connection to 127.18.222.61:38137: connect: Connection refused (error 111) I0918 20:39:17.654690 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:38137 I0918 20:39:17.656659 19320 webserver.cc:374] Webserver started at http://127.18.222.61:39575/ using document root and password file I0918 20:39:17.659049 25546 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:17.666184 25546 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:17.679457 25546 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:17.680285 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 156092us I0918 20:39:17.680773 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:17.681069 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:17.681403 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:17.681550 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:17.682598 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/master-2-root/instance: uuid: "d38eb92d8be54a50953dc5aed9dc8c62" format_stamp: "Formatted at 2019-09-18 20:39:17 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:17.687371 25489 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } attempt: 2 I0918 20:39:17.695621 25546 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:17.697965 25546 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:41641: Network error: Client connection negotiation failed: client connection to 127.18.222.60:41641: connect: Connection refused (error 111) I0918 20:39:17.699023 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.016s user 0.014s sys 0.000s I0918 20:39:17.701464 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:17.705034 25489 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:17.706686 25489 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:41641: Network error: Client connection negotiation failed: client connection to 127.18.222.60:41641: connect: Connection refused (error 111) I0918 20:39:17.710875 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:17.711127 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/master-2-root uuid: "d38eb92d8be54a50953dc5aed9dc8c62" format_stamp: "Formatted at 2019-09-18 20:39:17 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:17.711366 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:17.757155 25489 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } attempt: 1 I0918 20:39:17.758600 25546 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } attempt: 1 W0918 20:39:17.759616 25489 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:41641: Network error: Client connection negotiation failed: client connection to 127.18.222.60:41641: connect: Connection refused (error 111) W0918 20:39:17.760553 25546 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:41641: Network error: Client connection negotiation failed: client connection to 127.18.222.60:41641: connect: Connection refused (error 111) I0918 20:39:17.818143 25489 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } attempt: 2 W0918 20:39:17.820549 25489 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:41641: Network error: Client connection negotiation failed: client connection to 127.18.222.60:41641: connect: Connection refused (error 111) I0918 20:39:17.825639 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:41641 I0918 20:39:17.827512 19320 webserver.cc:374] Webserver started at http://127.18.222.60:46069/ using document root and password file I0918 20:39:17.827913 25546 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } attempt: 2 I0918 20:39:17.828627 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0918 20:39:17.829905 25604 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:17.837687 25604 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:17.848961 25546 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941: Bootstrap starting. I0918 20:39:17.852291 25604 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:17.852849 25546 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941: No blocks or log segments found. Creating new log. I0918 20:39:17.856596 25546 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941: No bootstrap required, opened a new log I0918 20:39:17.858567 25546 raft_consensus.cc:340] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } I0918 20:39:17.859103 25546 raft_consensus.cc:366] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:17.859285 25546 raft_consensus.cc:705] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f649e2399b743d9a53a42952539e941, State: Initialized, Role: FOLLOWER I0918 20:39:17.860108 25546 consensus_queue.cc:229] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } I0918 20:39:17.862172 25546 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } } I0918 20:39:17.862501 25608 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } } I0918 20:39:17.863023 25546 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:17.863271 25608 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:17.863929 25604 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:17.864362 25546 sys_catalog.cc:428] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941 [sys.catalog]: configured and running, proceeding with master startup. W0918 20:39:17.868173 25609 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:17.868386 25609 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:17.879482 25604 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62: Bootstrap starting. I0918 20:39:17.882874 25604 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62: No blocks or log segments found. Creating new log. I0918 20:39:17.886246 25604 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62: No bootstrap required, opened a new log I0918 20:39:17.888022 25604 raft_consensus.cc:340] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } I0918 20:39:17.888533 25604 raft_consensus.cc:366] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:17.888710 25604 raft_consensus.cc:705] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38eb92d8be54a50953dc5aed9dc8c62, State: Initialized, Role: FOLLOWER I0918 20:39:17.889257 25604 consensus_queue.cc:229] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } I0918 20:39:17.890830 25611 sys_catalog.cc:340] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } } I0918 20:39:17.891336 25611 sys_catalog.cc:343] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:17.891913 25604 sys_catalog.cc:340] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } } I0918 20:39:17.892452 25604 sys_catalog.cc:343] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:17.893373 25604 sys_catalog.cc:428] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:17.894385 25612 catalog_manager.cc:569] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0918 20:39:17.931151 25489 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } attempt: 3 I0918 20:39:17.946753 25489 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4: Bootstrap starting. I0918 20:39:17.949962 25489 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4: No blocks or log segments found. Creating new log. I0918 20:39:17.953495 25489 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4: No bootstrap required, opened a new log I0918 20:39:17.955193 25489 raft_consensus.cc:340] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } I0918 20:39:17.955819 25489 raft_consensus.cc:366] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:17.956025 25489 raft_consensus.cc:705] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8424e7d64a648f3952fd83c45ad7fc4, State: Initialized, Role: FOLLOWER I0918 20:39:17.956569 25489 consensus_queue.cc:229] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } I0918 20:39:17.958283 25489 sys_catalog.cc:340] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } } I0918 20:39:17.959030 25489 sys_catalog.cc:343] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:17.958568 25614 sys_catalog.cc:340] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } } I0918 20:39:17.959400 25614 sys_catalog.cc:343] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:17.960220 25489 sys_catalog.cc:428] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:17.961304 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 I0918 20:39:17.961513 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 W0918 20:39:17.963438 25615 catalog_manager.cc:1155] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:17.963766 25615 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:17.977556 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 156092us I0918 20:39:17.978055 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:17.978220 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:17.978483 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:17.978606 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:17.979635 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-0-root/instance: uuid: "c8bf460a227d45cabebe5bcdb8771950" format_stamp: "Formatted at 2019-09-18 20:39:17 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:17.989712 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.009s sys 0.000s I0918 20:39:17.991405 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:18.000210 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:39:18.000445 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-0-root uuid: "c8bf460a227d45cabebe5bcdb8771950" format_stamp: "Formatted at 2019-09-18 20:39:17 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:18.000727 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:18.019582 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:18.019774 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:18.039499 25611 raft_consensus.cc:472] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:18.040401 25611 raft_consensus.cc:494] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } I0918 20:39:18.042836 25611 leader_election.cc:296] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8424e7d64a648f3952fd83c45ad7fc4 (127.18.222.62:36069), 5f649e2399b743d9a53a42952539e941 (127.18.222.61:38137) I0918 20:39:18.043473 25466 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" is_pre_election: true I0918 20:39:18.043848 25523 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f649e2399b743d9a53a42952539e941" is_pre_election: true I0918 20:39:18.044340 25466 raft_consensus.cc:2350] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d38eb92d8be54a50953dc5aed9dc8c62 in term 0. I0918 20:39:18.044721 25523 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d38eb92d8be54a50953dc5aed9dc8c62 in term 0. I0918 20:39:18.045644 25554 leader_election.cc:310] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8424e7d64a648f3952fd83c45ad7fc4, d38eb92d8be54a50953dc5aed9dc8c62; no voters: I0918 20:39:18.046555 25611 raft_consensus.cc:2644] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:18.046923 25611 raft_consensus.cc:472] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:18.047242 25611 raft_consensus.cc:2890] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:18.055893 25611 raft_consensus.cc:494] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } I0918 20:39:18.058387 25611 leader_election.cc:296] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [CANDIDATE]: Term 1 election: Requested vote from peers b8424e7d64a648f3952fd83c45ad7fc4 (127.18.222.62:36069), 5f649e2399b743d9a53a42952539e941 (127.18.222.61:38137) I0918 20:39:18.059046 25466 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" I0918 20:39:18.059231 25523 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f649e2399b743d9a53a42952539e941" I0918 20:39:18.059599 25466 raft_consensus.cc:2890] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:18.060170 25523 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:18.066923 25523 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d38eb92d8be54a50953dc5aed9dc8c62 in term 1. I0918 20:39:18.068363 25466 raft_consensus.cc:2350] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d38eb92d8be54a50953dc5aed9dc8c62 in term 1. I0918 20:39:18.068603 25552 leader_election.cc:310] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f649e2399b743d9a53a42952539e941, d38eb92d8be54a50953dc5aed9dc8c62; no voters: I0918 20:39:18.069561 25611 raft_consensus.cc:2644] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:18.070950 25611 raft_consensus.cc:667] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [term 1 LEADER]: Becoming Leader. State: Replica: d38eb92d8be54a50953dc5aed9dc8c62, State: Running, Role: LEADER I0918 20:39:18.071900 25611 consensus_queue.cc:206] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } I0918 20:39:18.076092 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:45671 I0918 20:39:18.079002 25677 sys_catalog.cc:340] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d38eb92d8be54a50953dc5aed9dc8c62. Latest consensus state: current_term: 1 leader_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } } I0918 20:39:18.080013 25677 sys_catalog.cc:343] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [sys.catalog]: This master's current role is: LEADER I0918 20:39:18.081652 19320 webserver.cc:374] Webserver started at http://127.18.222.1:46097/ using document root and password file I0918 20:39:18.082685 25682 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:39:18.092998 25682 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:39:18.112807 25684 heartbeater.cc:322] Connected to a master server at 127.18.222.61:38137 I0918 20:39:18.113199 25684 heartbeater.cc:406] Registering TS with master... I0918 20:39:18.114567 25684 heartbeater.cc:455] Master 127.18.222.61:38137 requested a full tablet report, sending... I0918 20:39:18.117393 25513 ts_manager.cc:186] Registered new tserver with Master: c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671) I0918 20:39:18.122725 25683 heartbeater.cc:322] Connected to a master server at 127.18.222.60:41641 I0918 20:39:18.123073 25683 heartbeater.cc:406] Registering TS with master... I0918 20:39:18.123250 25686 heartbeater.cc:322] Connected to a master server at 127.18.222.62:36069 I0918 20:39:18.123653 25686 heartbeater.cc:406] Registering TS with master... I0918 20:39:18.123865 25683 heartbeater.cc:455] Master 127.18.222.60:41641 requested a full tablet report, sending... I0918 20:39:18.124195 25686 heartbeater.cc:455] Master 127.18.222.62:36069 requested a full tablet report, sending... I0918 20:39:18.126024 25571 ts_manager.cc:186] Registered new tserver with Master: c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671) I0918 20:39:18.126858 25456 ts_manager.cc:186] Registered new tserver with Master: c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671) I0918 20:39:18.129302 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 156592us I0918 20:39:18.129855 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:18.130085 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:18.130393 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:18.130578 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:18.132122 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-1-root/instance: uuid: "38427f7b42664adab4dfa31963a8b7c2" format_stamp: "Formatted at 2019-09-18 20:39:18 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:18.136355 25466 raft_consensus.cc:1184] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4 [term 1 FOLLOWER]: Refusing update from remote peer d38eb92d8be54a50953dc5aed9dc8c62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:18.136386 25523 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941 [term 1 FOLLOWER]: Refusing update from remote peer d38eb92d8be54a50953dc5aed9dc8c62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:18.138099 25611 consensus_queue.cc:984] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:18.139820 25677 consensus_queue.cc:984] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:18.149528 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.017s user 0.016s sys 0.000s I0918 20:39:18.152784 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:18.153033 25614 sys_catalog.cc:340] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d38eb92d8be54a50953dc5aed9dc8c62. Latest consensus state: current_term: 1 leader_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } } I0918 20:39:18.154088 25614 sys_catalog.cc:343] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:18.154013 25608 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d38eb92d8be54a50953dc5aed9dc8c62. Latest consensus state: current_term: 1 leader_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } } I0918 20:39:18.154739 25608 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:18.179311 25614 sys_catalog.cc:340] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } } I0918 20:39:18.180435 25614 sys_catalog.cc:343] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:18.184926 25611 sys_catalog.cc:340] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } } I0918 20:39:18.186861 25611 sys_catalog.cc:343] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [sys.catalog]: This master's current role is: LEADER I0918 20:39:18.188045 25611 sys_catalog.cc:340] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } } I0918 20:39:18.188993 25611 sys_catalog.cc:343] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [sys.catalog]: This master's current role is: LEADER I0918 20:39:18.191507 19320 fs_manager.cc:419] Time spent opening block manager: real 0.017s user 0.011s sys 0.004s I0918 20:39:18.192014 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-1-root uuid: "38427f7b42664adab4dfa31963a8b7c2" format_stamp: "Formatted at 2019-09-18 20:39:18 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:18.191615 25608 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8424e7d64a648f3952fd83c45ad7fc4" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 36069 } } peers { permanent_uuid: "5f649e2399b743d9a53a42952539e941" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38137 } } peers { permanent_uuid: "d38eb92d8be54a50953dc5aed9dc8c62" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41641 } } } I0918 20:39:18.193334 25608 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:18.192399 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:18.195596 25682 catalog_manager.cc:981] Generated new certificate authority record I0918 20:39:18.197996 25682 catalog_manager.cc:1106] Loading token signing keys... I0918 20:39:18.225986 25682 catalog_manager.cc:4446] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62: Generated new TSK 0 I0918 20:39:18.226826 25682 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:39:18.231868 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:18.232153 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:18.243607 25571 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57726 I0918 20:39:18.257938 25683 heartbeater.cc:447] Master 127.18.222.60:41641 was elected leader, sending a full tablet report... I0918 20:39:18.283321 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.2:37907 I0918 20:39:18.285444 19320 webserver.cc:374] Webserver started at http://127.18.222.2:37885/ using document root and password file I0918 20:39:18.317256 25763 heartbeater.cc:322] Connected to a master server at 127.18.222.60:41641 I0918 20:39:18.317839 25763 heartbeater.cc:406] Registering TS with master... I0918 20:39:18.326516 25763 heartbeater.cc:455] Master 127.18.222.60:41641 requested a full tablet report, sending... I0918 20:39:18.330425 25571 ts_manager.cc:186] Registered new tserver with Master: 38427f7b42664adab4dfa31963a8b7c2 (127.18.222.2:37907) I0918 20:39:18.332746 25571 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57732 I0918 20:39:18.341840 25764 heartbeater.cc:322] Connected to a master server at 127.18.222.61:38137 I0918 20:39:18.342238 25764 heartbeater.cc:406] Registering TS with master... I0918 20:39:18.343044 25764 heartbeater.cc:455] Master 127.18.222.61:38137 requested a full tablet report, sending... I0918 20:39:18.345628 25513 ts_manager.cc:186] Registered new tserver with Master: 38427f7b42664adab4dfa31963a8b7c2 (127.18.222.2:37907) I0918 20:39:18.348322 25766 heartbeater.cc:322] Connected to a master server at 127.18.222.62:36069 I0918 20:39:18.348644 25766 heartbeater.cc:406] Registering TS with master... I0918 20:39:18.348644 25763 heartbeater.cc:447] Master 127.18.222.60:41641 was elected leader, sending a full tablet report... I0918 20:39:18.352257 25766 heartbeater.cc:455] Master 127.18.222.62:36069 requested a full tablet report, sending... I0918 20:39:18.354122 25455 ts_manager.cc:186] Registered new tserver with Master: 38427f7b42664adab4dfa31963a8b7c2 (127.18.222.2:37907) I0918 20:39:18.357429 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 156592us I0918 20:39:18.357918 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:18.358129 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:18.358387 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:18.358577 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:18.359972 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-2-root/instance: uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" format_stamp: "Formatted at 2019-09-18 20:39:18 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:18.374009 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.013s user 0.013s sys 0.000s I0918 20:39:18.376289 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:18.386248 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:39:18.386533 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-2-root uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" format_stamp: "Formatted at 2019-09-18 20:39:18 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:18.386799 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:18.419217 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:18.419500 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:18.479599 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.3:46373 I0918 20:39:18.481701 19320 webserver.cc:374] Webserver started at http://127.18.222.3:45035/ using document root and password file I0918 20:39:18.548889 25835 heartbeater.cc:322] Connected to a master server at 127.18.222.61:38137 I0918 20:39:18.549373 25835 heartbeater.cc:406] Registering TS with master... I0918 20:39:18.549957 25835 heartbeater.cc:455] Master 127.18.222.61:38137 requested a full tablet report, sending... I0918 20:39:18.552646 25513 ts_manager.cc:186] Registered new tserver with Master: 7d9b076d29dc4a24860e3f89aec6b4a0 (127.18.222.3:46373) I0918 20:39:18.559962 25834 heartbeater.cc:322] Connected to a master server at 127.18.222.60:41641 I0918 20:39:18.560253 25834 heartbeater.cc:406] Registering TS with master... I0918 20:39:18.560834 25834 heartbeater.cc:455] Master 127.18.222.60:41641 requested a full tablet report, sending... I0918 20:39:18.561087 25837 heartbeater.cc:322] Connected to a master server at 127.18.222.62:36069 I0918 20:39:18.561436 25837 heartbeater.cc:406] Registering TS with master... I0918 20:39:18.561959 25837 heartbeater.cc:455] Master 127.18.222.62:36069 requested a full tablet report, sending... I0918 20:39:18.562440 25570 ts_manager.cc:186] Registered new tserver with Master: 7d9b076d29dc4a24860e3f89aec6b4a0 (127.18.222.3:46373) I0918 20:39:18.564796 25570 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57738 I0918 20:39:18.564949 25455 ts_manager.cc:186] Registered new tserver with Master: 7d9b076d29dc4a24860e3f89aec6b4a0 (127.18.222.3:46373) I0918 20:39:18.567631 19320 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.026528791s I0918 20:39:18.580261 25834 heartbeater.cc:447] Master 127.18.222.60:41641 was elected leader, sending a full tablet report... I0918 20:39:18.629417 25571 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57748: name: "ksck-test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\n\000\000\000\004\001\024\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:39:18.631361 25571 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:39:18.750808 25800 tablet_service.cc:1022] Processing CreateTablet for tablet d1c2d835755e4978a9a6dde379385a16 (table=ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:18.752179 25800 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet d1c2d835755e4978a9a6dde379385a16. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:18.751062 25801 tablet_service.cc:1022] Processing CreateTablet for tablet ea2d24e8ebda434fa71708bf722f2054 (table=ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:18.753154 25801 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet ea2d24e8ebda434fa71708bf722f2054. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:18.751926 25799 tablet_service.cc:1022] Processing CreateTablet for tablet de8606c55a2e478bb968edb59bf5140f (table=ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:18.757503 25799 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet de8606c55a2e478bb968edb59bf5140f. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:18.759784 25645 tablet_service.cc:1022] Processing CreateTablet for tablet de8606c55a2e478bb968edb59bf5140f (table=ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:18.760426 25646 tablet_service.cc:1022] Processing CreateTablet for tablet d1c2d835755e4978a9a6dde379385a16 (table=ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:18.760947 25645 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet de8606c55a2e478bb968edb59bf5140f. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:18.761385 25646 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet d1c2d835755e4978a9a6dde379385a16. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:18.765957 25647 tablet_service.cc:1022] Processing CreateTablet for tablet ea2d24e8ebda434fa71708bf722f2054 (table=ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:18.767048 25647 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet ea2d24e8ebda434fa71708bf722f2054. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:18.790901 25730 tablet_service.cc:1022] Processing CreateTablet for tablet ea2d24e8ebda434fa71708bf722f2054 (table=ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:18.792330 25730 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet ea2d24e8ebda434fa71708bf722f2054. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:18.794123 25729 tablet_service.cc:1022] Processing CreateTablet for tablet d1c2d835755e4978a9a6dde379385a16 (table=ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:18.795208 25729 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet d1c2d835755e4978a9a6dde379385a16. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:18.804666 25728 tablet_service.cc:1022] Processing CreateTablet for tablet de8606c55a2e478bb968edb59bf5140f (table=ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:18.805838 25728 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet de8606c55a2e478bb968edb59bf5140f. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:18.833684 25855 tablet_bootstrap.cc:439] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0: Bootstrap starting. I0918 20:39:18.837617 25855 tablet_bootstrap.cc:590] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0: No blocks or log segments found. Creating new log. I0918 20:39:18.845654 25856 tablet_bootstrap.cc:439] T de8606c55a2e478bb968edb59bf5140f P c8bf460a227d45cabebe5bcdb8771950: Bootstrap starting. I0918 20:39:18.859644 25857 tablet_bootstrap.cc:439] T d1c2d835755e4978a9a6dde379385a16 P 38427f7b42664adab4dfa31963a8b7c2: Bootstrap starting. I0918 20:39:18.862799 25857 tablet_bootstrap.cc:590] T d1c2d835755e4978a9a6dde379385a16 P 38427f7b42664adab4dfa31963a8b7c2: No blocks or log segments found. Creating new log. I0918 20:39:18.866645 25856 tablet_bootstrap.cc:590] T de8606c55a2e478bb968edb59bf5140f P c8bf460a227d45cabebe5bcdb8771950: No blocks or log segments found. Creating new log. I0918 20:39:18.889956 25609 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941: acquiring CA information for follower catalog manager: success I0918 20:39:18.891672 25855 tablet_bootstrap.cc:439] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0: No bootstrap required, opened a new log I0918 20:39:18.894517 25855 ts_tablet_manager.cc:1085] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0: Time spent bootstrapping tablet: real 0.093s user 0.006s sys 0.005s I0918 20:39:18.897078 25855 raft_consensus.cc:340] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:18.897840 25855 raft_consensus.cc:366] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:18.898073 25855 raft_consensus.cc:705] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9b076d29dc4a24860e3f89aec6b4a0, State: Initialized, Role: FOLLOWER I0918 20:39:18.898854 25855 consensus_queue.cc:229] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:18.905795 25857 tablet_bootstrap.cc:439] T d1c2d835755e4978a9a6dde379385a16 P 38427f7b42664adab4dfa31963a8b7c2: No bootstrap required, opened a new log I0918 20:39:18.911250 25857 ts_tablet_manager.cc:1085] T d1c2d835755e4978a9a6dde379385a16 P 38427f7b42664adab4dfa31963a8b7c2: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.006s I0918 20:39:18.912415 25855 ts_tablet_manager.cc:1113] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0: Time spent starting tablet: real 0.018s user 0.013s sys 0.000s I0918 20:39:18.913499 25855 tablet_bootstrap.cc:439] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0: Bootstrap starting. I0918 20:39:18.914683 25856 tablet_bootstrap.cc:439] T de8606c55a2e478bb968edb59bf5140f P c8bf460a227d45cabebe5bcdb8771950: No bootstrap required, opened a new log I0918 20:39:18.915225 25856 ts_tablet_manager.cc:1085] T de8606c55a2e478bb968edb59bf5140f P c8bf460a227d45cabebe5bcdb8771950: Time spent bootstrapping tablet: real 0.070s user 0.012s sys 0.009s I0918 20:39:18.917991 25856 raft_consensus.cc:340] T de8606c55a2e478bb968edb59bf5140f P c8bf460a227d45cabebe5bcdb8771950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:18.918835 25856 raft_consensus.cc:366] T de8606c55a2e478bb968edb59bf5140f P c8bf460a227d45cabebe5bcdb8771950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:18.919102 25856 raft_consensus.cc:705] T de8606c55a2e478bb968edb59bf5140f P c8bf460a227d45cabebe5bcdb8771950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8bf460a227d45cabebe5bcdb8771950, State: Initialized, Role: FOLLOWER I0918 20:39:18.919950 25856 consensus_queue.cc:229] T de8606c55a2e478bb968edb59bf5140f P c8bf460a227d45cabebe5bcdb8771950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:18.922098 25857 raft_consensus.cc:340] T d1c2d835755e4978a9a6dde379385a16 P 38427f7b42664adab4dfa31963a8b7c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:18.923004 25857 raft_consensus.cc:366] T d1c2d835755e4978a9a6dde379385a16 P 38427f7b42664adab4dfa31963a8b7c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:18.923282 25857 raft_consensus.cc:705] T d1c2d835755e4978a9a6dde379385a16 P 38427f7b42664adab4dfa31963a8b7c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38427f7b42664adab4dfa31963a8b7c2, State: Initialized, Role: FOLLOWER I0918 20:39:18.923341 25855 tablet_bootstrap.cc:590] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0: No blocks or log segments found. Creating new log. I0918 20:39:18.924172 25857 consensus_queue.cc:229] T d1c2d835755e4978a9a6dde379385a16 P 38427f7b42664adab4dfa31963a8b7c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:18.920691 25609 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:18.934810 25856 ts_tablet_manager.cc:1113] T de8606c55a2e478bb968edb59bf5140f P c8bf460a227d45cabebe5bcdb8771950: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s I0918 20:39:18.936880 25856 tablet_bootstrap.cc:439] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950: Bootstrap starting. I0918 20:39:18.946645 25856 tablet_bootstrap.cc:590] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950: No blocks or log segments found. Creating new log. I0918 20:39:18.948969 25855 tablet_bootstrap.cc:439] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0: No bootstrap required, opened a new log I0918 20:39:18.962056 25855 ts_tablet_manager.cc:1085] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.005s I0918 20:39:18.954592 25857 ts_tablet_manager.cc:1113] T d1c2d835755e4978a9a6dde379385a16 P 38427f7b42664adab4dfa31963a8b7c2: Time spent starting tablet: real 0.039s user 0.003s sys 0.003s I0918 20:39:18.965018 25857 tablet_bootstrap.cc:439] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2: Bootstrap starting. I0918 20:39:18.967557 25857 tablet_bootstrap.cc:590] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2: No blocks or log segments found. Creating new log. I0918 20:39:18.968356 25855 raft_consensus.cc:340] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:18.969162 25855 raft_consensus.cc:366] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:18.969364 25855 raft_consensus.cc:705] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9b076d29dc4a24860e3f89aec6b4a0, State: Initialized, Role: FOLLOWER I0918 20:39:18.969841 25615 catalog_manager.cc:1153] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4: acquiring CA information for follower catalog manager: success I0918 20:39:18.976884 25856 tablet_bootstrap.cc:439] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950: No bootstrap required, opened a new log I0918 20:39:18.977517 25856 ts_tablet_manager.cc:1085] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.007s I0918 20:39:18.978828 25615 catalog_manager.cc:1181] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:18.979859 25856 raft_consensus.cc:340] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:18.980805 25856 raft_consensus.cc:366] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:18.969914 25855 consensus_queue.cc:229] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:18.981137 25856 raft_consensus.cc:705] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8bf460a227d45cabebe5bcdb8771950, State: Initialized, Role: FOLLOWER I0918 20:39:18.982038 25856 consensus_queue.cc:229] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:18.982697 25857 tablet_bootstrap.cc:439] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2: No bootstrap required, opened a new log I0918 20:39:18.983458 25857 ts_tablet_manager.cc:1085] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.001s I0918 20:39:18.985888 25855 ts_tablet_manager.cc:1113] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0: Time spent starting tablet: real 0.023s user 0.004s sys 0.000s I0918 20:39:18.989570 25857 raft_consensus.cc:340] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:18.990206 25857 raft_consensus.cc:366] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:18.990375 25855 tablet_bootstrap.cc:439] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0: Bootstrap starting. I0918 20:39:18.990595 25857 raft_consensus.cc:705] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38427f7b42664adab4dfa31963a8b7c2, State: Initialized, Role: FOLLOWER I0918 20:39:18.991417 25857 consensus_queue.cc:229] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:18.994253 25855 tablet_bootstrap.cc:590] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0: No blocks or log segments found. Creating new log. I0918 20:39:19.002179 25857 ts_tablet_manager.cc:1113] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s I0918 20:39:19.003070 25857 tablet_bootstrap.cc:439] T de8606c55a2e478bb968edb59bf5140f P 38427f7b42664adab4dfa31963a8b7c2: Bootstrap starting. I0918 20:39:19.006280 25855 tablet_bootstrap.cc:439] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0: No bootstrap required, opened a new log I0918 20:39:19.006909 25855 ts_tablet_manager.cc:1085] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.004s I0918 20:39:19.010123 25861 raft_consensus.cc:472] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:19.010707 25857 tablet_bootstrap.cc:590] T de8606c55a2e478bb968edb59bf5140f P 38427f7b42664adab4dfa31963a8b7c2: No blocks or log segments found. Creating new log. I0918 20:39:19.010726 25861 raft_consensus.cc:494] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:19.013572 25861 leader_election.cc:296] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671), 38427f7b42664adab4dfa31963a8b7c2 (127.18.222.2:37907) I0918 20:39:19.021013 25855 raft_consensus.cc:340] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:19.021612 25856 ts_tablet_manager.cc:1113] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950: Time spent starting tablet: real 0.044s user 0.006s sys 0.001s I0918 20:39:19.022536 25856 tablet_bootstrap.cc:439] T ea2d24e8ebda434fa71708bf722f2054 P c8bf460a227d45cabebe5bcdb8771950: Bootstrap starting. I0918 20:39:19.025421 25855 raft_consensus.cc:366] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:19.025756 25855 raft_consensus.cc:705] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9b076d29dc4a24860e3f89aec6b4a0, State: Initialized, Role: FOLLOWER I0918 20:39:19.026052 25856 tablet_bootstrap.cc:590] T ea2d24e8ebda434fa71708bf722f2054 P c8bf460a227d45cabebe5bcdb8771950: No blocks or log segments found. Creating new log. I0918 20:39:19.026554 25855 consensus_queue.cc:229] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:19.031993 25861 raft_consensus.cc:472] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:19.032472 25861 raft_consensus.cc:494] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:19.040046 25861 leader_election.cc:296] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671), 38427f7b42664adab4dfa31963a8b7c2 (127.18.222.2:37907) W0918 20:39:19.046232 25838 tablet.cc:1763] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0: Can't schedule compaction. Clean time has not been advanced past its initial value. I0918 20:39:19.105937 25855 ts_tablet_manager.cc:1113] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0: Time spent starting tablet: real 0.099s user 0.007s sys 0.001s I0918 20:39:19.106858 25864 raft_consensus.cc:472] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:19.107408 25864 raft_consensus.cc:494] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:19.114361 25857 tablet_bootstrap.cc:439] T de8606c55a2e478bb968edb59bf5140f P 38427f7b42664adab4dfa31963a8b7c2: No bootstrap required, opened a new log I0918 20:39:19.115074 25857 ts_tablet_manager.cc:1085] T de8606c55a2e478bb968edb59bf5140f P 38427f7b42664adab4dfa31963a8b7c2: Time spent bootstrapping tablet: real 0.112s user 0.028s sys 0.022s I0918 20:39:19.118278 25857 raft_consensus.cc:340] T de8606c55a2e478bb968edb59bf5140f P 38427f7b42664adab4dfa31963a8b7c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:19.119151 25857 raft_consensus.cc:366] T de8606c55a2e478bb968edb59bf5140f P 38427f7b42664adab4dfa31963a8b7c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:19.122740 25857 raft_consensus.cc:705] T de8606c55a2e478bb968edb59bf5140f P 38427f7b42664adab4dfa31963a8b7c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38427f7b42664adab4dfa31963a8b7c2, State: Initialized, Role: FOLLOWER I0918 20:39:19.120548 25862 raft_consensus.cc:472] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:19.125188 25864 leader_election.cc:296] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671), 7d9b076d29dc4a24860e3f89aec6b4a0 (127.18.222.3:46373) I0918 20:39:19.149221 25740 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d1c2d835755e4978a9a6dde379385a16" candidate_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38427f7b42664adab4dfa31963a8b7c2" is_pre_election: true I0918 20:39:19.150213 25740 raft_consensus.cc:2350] T d1c2d835755e4978a9a6dde379385a16 P 38427f7b42664adab4dfa31963a8b7c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d9b076d29dc4a24860e3f89aec6b4a0 in term 0. I0918 20:39:19.169239 25657 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d1c2d835755e4978a9a6dde379385a16" candidate_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8bf460a227d45cabebe5bcdb8771950" is_pre_election: true I0918 20:39:19.176606 25656 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ea2d24e8ebda434fa71708bf722f2054" candidate_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8bf460a227d45cabebe5bcdb8771950" is_pre_election: true I0918 20:39:19.142894 25862 raft_consensus.cc:494] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:19.142251 25857 consensus_queue.cc:229] T de8606c55a2e478bb968edb59bf5140f P 38427f7b42664adab4dfa31963a8b7c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:19.188465 25657 raft_consensus.cc:2350] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d9b076d29dc4a24860e3f89aec6b4a0 in term 0. W0918 20:39:19.187340 25774 leader_election.cc:348] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671): Illegal state: must be running to vote when last-logged opid is not known I0918 20:39:19.202807 25862 leader_election.cc:296] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38427f7b42664adab4dfa31963a8b7c2 (127.18.222.2:37907), 7d9b076d29dc4a24860e3f89aec6b4a0 (127.18.222.3:46373) I0918 20:39:19.195636 25774 leader_election.cc:310] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d9b076d29dc4a24860e3f89aec6b4a0, c8bf460a227d45cabebe5bcdb8771950; no voters: I0918 20:39:19.204219 25869 raft_consensus.cc:2644] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:19.205931 25869 raft_consensus.cc:472] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:19.211213 25869 raft_consensus.cc:2890] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:19.230147 25857 ts_tablet_manager.cc:1113] T de8606c55a2e478bb968edb59bf5140f P 38427f7b42664adab4dfa31963a8b7c2: Time spent starting tablet: real 0.115s user 0.002s sys 0.005s I0918 20:39:19.239419 25861 raft_consensus.cc:472] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:19.240111 25861 raft_consensus.cc:494] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:19.241974 25861 leader_election.cc:296] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671), 38427f7b42664adab4dfa31963a8b7c2 (127.18.222.2:37907) I0918 20:39:19.225960 25869 raft_consensus.cc:494] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:19.261323 25856 tablet_bootstrap.cc:439] T ea2d24e8ebda434fa71708bf722f2054 P c8bf460a227d45cabebe5bcdb8771950: No bootstrap required, opened a new log I0918 20:39:19.263103 25657 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de8606c55a2e478bb968edb59bf5140f" candidate_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8bf460a227d45cabebe5bcdb8771950" is_pre_election: true I0918 20:39:19.264483 25657 raft_consensus.cc:2350] T de8606c55a2e478bb968edb59bf5140f P c8bf460a227d45cabebe5bcdb8771950 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d9b076d29dc4a24860e3f89aec6b4a0 in term 0. I0918 20:39:19.267210 25740 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ea2d24e8ebda434fa71708bf722f2054" candidate_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38427f7b42664adab4dfa31963a8b7c2" is_pre_election: true I0918 20:39:19.268023 25740 raft_consensus.cc:2350] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d9b076d29dc4a24860e3f89aec6b4a0 in term 0. I0918 20:39:19.263674 25856 ts_tablet_manager.cc:1085] T ea2d24e8ebda434fa71708bf722f2054 P c8bf460a227d45cabebe5bcdb8771950: Time spent bootstrapping tablet: real 0.241s user 0.020s sys 0.028s I0918 20:39:19.273741 25657 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d1c2d835755e4978a9a6dde379385a16" candidate_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8bf460a227d45cabebe5bcdb8771950" I0918 20:39:19.274435 25657 raft_consensus.cc:2890] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:19.277101 25740 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de8606c55a2e478bb968edb59bf5140f" candidate_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38427f7b42664adab4dfa31963a8b7c2" is_pre_election: true I0918 20:39:19.277820 25740 raft_consensus.cc:2350] T de8606c55a2e478bb968edb59bf5140f P 38427f7b42664adab4dfa31963a8b7c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d9b076d29dc4a24860e3f89aec6b4a0 in term 0. I0918 20:39:19.281235 25657 raft_consensus.cc:2350] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d9b076d29dc4a24860e3f89aec6b4a0 in term 1. I0918 20:39:19.282128 25740 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d1c2d835755e4978a9a6dde379385a16" candidate_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38427f7b42664adab4dfa31963a8b7c2" I0918 20:39:19.283529 25869 leader_election.cc:296] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 [CANDIDATE]: Term 1 election: Requested vote from peers c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671), 38427f7b42664adab4dfa31963a8b7c2 (127.18.222.2:37907) I0918 20:39:19.283910 25774 leader_election.cc:310] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d9b076d29dc4a24860e3f89aec6b4a0, c8bf460a227d45cabebe5bcdb8771950; no voters: I0918 20:39:19.292559 25774 leader_election.cc:310] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38427f7b42664adab4dfa31963a8b7c2, 7d9b076d29dc4a24860e3f89aec6b4a0; no voters: c8bf460a227d45cabebe5bcdb8771950 I0918 20:39:19.293423 25861 raft_consensus.cc:2644] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:19.293745 25861 raft_consensus.cc:472] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:19.294029 25861 raft_consensus.cc:2890] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:19.288650 25740 raft_consensus.cc:2890] T d1c2d835755e4978a9a6dde379385a16 P 38427f7b42664adab4dfa31963a8b7c2 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:19.300987 25861 raft_consensus.cc:494] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:19.303088 25657 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ea2d24e8ebda434fa71708bf722f2054" candidate_uuid: "38427f7b42664adab4dfa31963a8b7c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8bf460a227d45cabebe5bcdb8771950" is_pre_election: true I0918 20:39:19.304014 25739 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de8606c55a2e478bb968edb59bf5140f" candidate_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38427f7b42664adab4dfa31963a8b7c2" I0918 20:39:19.304481 25739 raft_consensus.cc:2890] T de8606c55a2e478bb968edb59bf5140f P 38427f7b42664adab4dfa31963a8b7c2 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:19.309373 25739 raft_consensus.cc:2350] T de8606c55a2e478bb968edb59bf5140f P 38427f7b42664adab4dfa31963a8b7c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d9b076d29dc4a24860e3f89aec6b4a0 in term 1. I0918 20:39:19.310937 25740 raft_consensus.cc:2350] T d1c2d835755e4978a9a6dde379385a16 P 38427f7b42664adab4dfa31963a8b7c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d9b076d29dc4a24860e3f89aec6b4a0 in term 1. I0918 20:39:19.313850 25774 leader_election.cc:310] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38427f7b42664adab4dfa31963a8b7c2, 7d9b076d29dc4a24860e3f89aec6b4a0; no voters: I0918 20:39:19.315883 25656 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de8606c55a2e478bb968edb59bf5140f" candidate_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8bf460a227d45cabebe5bcdb8771950" I0918 20:39:19.316555 25656 raft_consensus.cc:2890] T de8606c55a2e478bb968edb59bf5140f P c8bf460a227d45cabebe5bcdb8771950 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:19.318251 25861 leader_election.cc:296] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 [CANDIDATE]: Term 1 election: Requested vote from peers c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671), 38427f7b42664adab4dfa31963a8b7c2 (127.18.222.2:37907) I0918 20:39:19.319033 25861 raft_consensus.cc:2644] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:19.319347 25861 raft_consensus.cc:472] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:19.319646 25861 raft_consensus.cc:2890] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:19.324785 25656 raft_consensus.cc:2350] T de8606c55a2e478bb968edb59bf5140f P c8bf460a227d45cabebe5bcdb8771950 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d9b076d29dc4a24860e3f89aec6b4a0 in term 1. I0918 20:39:19.327849 25811 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d1c2d835755e4978a9a6dde379385a16" candidate_uuid: "c8bf460a227d45cabebe5bcdb8771950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" is_pre_election: true I0918 20:39:19.289270 25856 raft_consensus.cc:340] T ea2d24e8ebda434fa71708bf722f2054 P c8bf460a227d45cabebe5bcdb8771950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:19.328866 25811 raft_consensus.cc:2280] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c8bf460a227d45cabebe5bcdb8771950 in current term 1: Already voted for candidate 7d9b076d29dc4a24860e3f89aec6b4a0 in this term. I0918 20:39:19.329632 25856 raft_consensus.cc:366] T ea2d24e8ebda434fa71708bf722f2054 P c8bf460a227d45cabebe5bcdb8771950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:19.329906 25856 raft_consensus.cc:705] T ea2d24e8ebda434fa71708bf722f2054 P c8bf460a227d45cabebe5bcdb8771950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8bf460a227d45cabebe5bcdb8771950, State: Initialized, Role: FOLLOWER I0918 20:39:19.330673 25740 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d1c2d835755e4978a9a6dde379385a16" candidate_uuid: "c8bf460a227d45cabebe5bcdb8771950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38427f7b42664adab4dfa31963a8b7c2" is_pre_election: true I0918 20:39:19.330565 25856 consensus_queue.cc:229] T ea2d24e8ebda434fa71708bf722f2054 P c8bf460a227d45cabebe5bcdb8771950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:19.331223 25740 raft_consensus.cc:2280] T d1c2d835755e4978a9a6dde379385a16 P 38427f7b42664adab4dfa31963a8b7c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c8bf460a227d45cabebe5bcdb8771950 in current term 1: Already voted for candidate 7d9b076d29dc4a24860e3f89aec6b4a0 in this term. I0918 20:39:19.331538 25657 raft_consensus.cc:2350] T ea2d24e8ebda434fa71708bf722f2054 P c8bf460a227d45cabebe5bcdb8771950 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38427f7b42664adab4dfa31963a8b7c2 in term 0. I0918 20:39:19.333176 25703 leader_election.cc:310] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38427f7b42664adab4dfa31963a8b7c2, c8bf460a227d45cabebe5bcdb8771950; no voters: I0918 20:39:19.334931 25861 raft_consensus.cc:494] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:19.340024 25864 raft_consensus.cc:2644] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:19.340354 25864 raft_consensus.cc:472] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:19.340631 25864 raft_consensus.cc:2890] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:19.344552 25774 leader_election.cc:310] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d9b076d29dc4a24860e3f89aec6b4a0, c8bf460a227d45cabebe5bcdb8771950; no voters: I0918 20:39:19.327965 25869 raft_consensus.cc:2644] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:19.339412 25856 ts_tablet_manager.cc:1113] T ea2d24e8ebda434fa71708bf722f2054 P c8bf460a227d45cabebe5bcdb8771950: Time spent starting tablet: real 0.071s user 0.006s sys 0.000s I0918 20:39:19.348266 25864 raft_consensus.cc:494] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:19.349005 25861 leader_election.cc:296] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0 [CANDIDATE]: Term 1 election: Requested vote from peers c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671), 38427f7b42664adab4dfa31963a8b7c2 (127.18.222.2:37907) I0918 20:39:19.349586 25861 raft_consensus.cc:2644] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:19.349978 25861 raft_consensus.cc:667] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 1 LEADER]: Becoming Leader. State: Replica: 7d9b076d29dc4a24860e3f89aec6b4a0, State: Running, Role: LEADER I0918 20:39:19.350003 25864 leader_election.cc:296] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [CANDIDATE]: Term 1 election: Requested vote from peers c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671), 7d9b076d29dc4a24860e3f89aec6b4a0 (127.18.222.3:46373) I0918 20:39:19.350562 25861 consensus_queue.cc:206] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:19.362155 25620 leader_election.cc:310] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8bf460a227d45cabebe5bcdb8771950; no voters: 38427f7b42664adab4dfa31963a8b7c2, 7d9b076d29dc4a24860e3f89aec6b4a0 I0918 20:39:19.358989 25869 raft_consensus.cc:667] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 1 LEADER]: Becoming Leader. State: Replica: 7d9b076d29dc4a24860e3f89aec6b4a0, State: Running, Role: LEADER I0918 20:39:19.362524 25656 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ea2d24e8ebda434fa71708bf722f2054" candidate_uuid: "38427f7b42664adab4dfa31963a8b7c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8bf460a227d45cabebe5bcdb8771950" I0918 20:39:19.362766 25657 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ea2d24e8ebda434fa71708bf722f2054" candidate_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8bf460a227d45cabebe5bcdb8771950" I0918 20:39:19.363210 25656 raft_consensus.cc:2890] T ea2d24e8ebda434fa71708bf722f2054 P c8bf460a227d45cabebe5bcdb8771950 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:19.364320 25862 raft_consensus.cc:2592] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0918 20:39:19.371793 25740 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ea2d24e8ebda434fa71708bf722f2054" candidate_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38427f7b42664adab4dfa31963a8b7c2" I0918 20:39:19.372010 25656 raft_consensus.cc:2350] T ea2d24e8ebda434fa71708bf722f2054 P c8bf460a227d45cabebe5bcdb8771950 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38427f7b42664adab4dfa31963a8b7c2 in term 1. I0918 20:39:19.372597 25740 raft_consensus.cc:2280] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7d9b076d29dc4a24860e3f89aec6b4a0 in current term 1: Already voted for candidate 38427f7b42664adab4dfa31963a8b7c2 in this term. I0918 20:39:19.372658 25657 raft_consensus.cc:2325] T ea2d24e8ebda434fa71708bf722f2054 P c8bf460a227d45cabebe5bcdb8771950 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7d9b076d29dc4a24860e3f89aec6b4a0 for term 1 because replica is already servicing an update from a current leader or another vote. I0918 20:39:19.373389 25703 leader_election.cc:310] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38427f7b42664adab4dfa31963a8b7c2, c8bf460a227d45cabebe5bcdb8771950; no voters: I0918 20:39:19.374439 25864 raft_consensus.cc:2644] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:19.363095 25869 consensus_queue.cc:206] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:19.374851 25774 leader_election.cc:310] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d9b076d29dc4a24860e3f89aec6b4a0; no voters: 38427f7b42664adab4dfa31963a8b7c2, c8bf460a227d45cabebe5bcdb8771950 I0918 20:39:19.376359 25878 raft_consensus.cc:2592] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority I0918 20:39:19.377701 25864 raft_consensus.cc:667] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [term 1 LEADER]: Becoming Leader. State: Replica: 38427f7b42664adab4dfa31963a8b7c2, State: Running, Role: LEADER I0918 20:39:19.378428 25864 consensus_queue.cc:206] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } } I0918 20:39:19.381938 25811 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ea2d24e8ebda434fa71708bf722f2054" candidate_uuid: "38427f7b42664adab4dfa31963a8b7c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" is_pre_election: true I0918 20:39:19.382856 25811 raft_consensus.cc:2280] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38427f7b42664adab4dfa31963a8b7c2 in current term 1: Already voted for candidate 7d9b076d29dc4a24860e3f89aec6b4a0 in this term. I0918 20:39:19.384480 25811 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ea2d24e8ebda434fa71708bf722f2054" candidate_uuid: "38427f7b42664adab4dfa31963a8b7c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" I0918 20:39:19.386590 25811 raft_consensus.cc:2280] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 38427f7b42664adab4dfa31963a8b7c2 in current term 1: Already voted for candidate 7d9b076d29dc4a24860e3f89aec6b4a0 in this term. I0918 20:39:19.389963 25571 catalog_manager.cc:4131] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 reported cstate change: term changed from 0 to 1, leader changed from to 38427f7b42664adab4dfa31963a8b7c2 (127.18.222.2). New cstate: current_term: 1 leader_uuid: "38427f7b42664adab4dfa31963a8b7c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } health_report { overall_health: UNKNOWN } } } I0918 20:39:19.398332 25568 catalog_manager.cc:4131] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 reported cstate change: term changed from 0 to 1, leader changed from to 7d9b076d29dc4a24860e3f89aec6b4a0 (127.18.222.3). New cstate: current_term: 1 leader_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } health_report { overall_health: HEALTHY } } } I0918 20:39:19.400311 25568 catalog_manager.cc:4131] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 reported cstate change: term changed from 0 to 1, leader changed from to 7d9b076d29dc4a24860e3f89aec6b4a0 (127.18.222.3). New cstate: current_term: 1 leader_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d9b076d29dc4a24860e3f89aec6b4a0" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 46373 } health_report { overall_health: HEALTHY } } } I0918 20:39:19.528539 25570 catalog_manager.cc:1947] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57748: table { table_name: "ksck-test-table" } modify_external_catalogs: true I0918 20:39:19.560451 25570 catalog_manager.cc:4381] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62: Sending DeleteTablet for 3 replicas of tablet ea2d24e8ebda434fa71708bf722f2054 I0918 20:39:19.561077 25570 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ea2d24e8ebda434fa71708bf722f2054 on c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671) (Table deleted at 2019-09-18 20:39:19 UTC) I0918 20:39:19.562705 25647 tablet_service.cc:1066] Processing DeleteTablet for tablet ea2d24e8ebda434fa71708bf722f2054 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-18 20:39:19 UTC) from {username='slave'} at 127.0.0.1:54632 I0918 20:39:19.564129 25570 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ea2d24e8ebda434fa71708bf722f2054 on 38427f7b42664adab4dfa31963a8b7c2 (127.18.222.2:37907) (Table deleted at 2019-09-18 20:39:19 UTC) I0918 20:39:19.565687 25570 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ea2d24e8ebda434fa71708bf722f2054 on 7d9b076d29dc4a24860e3f89aec6b4a0 (127.18.222.3:46373) (Table deleted at 2019-09-18 20:39:19 UTC) I0918 20:39:19.566323 25570 catalog_manager.cc:4381] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62: Sending DeleteTablet for 3 replicas of tablet d1c2d835755e4978a9a6dde379385a16 I0918 20:39:19.568514 25728 tablet_service.cc:1066] Processing DeleteTablet for tablet ea2d24e8ebda434fa71708bf722f2054 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-18 20:39:19 UTC) from {username='slave'} at 127.0.0.1:57826 I0918 20:39:19.574748 25570 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d1c2d835755e4978a9a6dde379385a16 on c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671) (Table deleted at 2019-09-18 20:39:19 UTC) I0918 20:39:19.575911 25799 tablet_service.cc:1066] Processing DeleteTablet for tablet ea2d24e8ebda434fa71708bf722f2054 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-18 20:39:19 UTC) from {username='slave'} at 127.0.0.1:45926 I0918 20:39:19.576038 25897 tablet_replica.cc:273] T ea2d24e8ebda434fa71708bf722f2054 P c8bf460a227d45cabebe5bcdb8771950: stopping tablet replica I0918 20:39:19.577327 25647 tablet_service.cc:1066] Processing DeleteTablet for tablet d1c2d835755e4978a9a6dde379385a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-18 20:39:19 UTC) from {username='slave'} at 127.0.0.1:54632 I0918 20:39:19.577440 25570 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d1c2d835755e4978a9a6dde379385a16 on 38427f7b42664adab4dfa31963a8b7c2 (127.18.222.2:37907) (Table deleted at 2019-09-18 20:39:19 UTC) I0918 20:39:19.576591 25898 tablet_replica.cc:273] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2: stopping tablet replica I0918 20:39:19.583027 25898 raft_consensus.cc:2147] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:19.581374 25897 raft_consensus.cc:2147] T ea2d24e8ebda434fa71708bf722f2054 P c8bf460a227d45cabebe5bcdb8771950 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:19.586334 25897 raft_consensus.cc:2174] T ea2d24e8ebda434fa71708bf722f2054 P c8bf460a227d45cabebe5bcdb8771950 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:19.587649 25728 tablet_service.cc:1066] Processing DeleteTablet for tablet d1c2d835755e4978a9a6dde379385a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-18 20:39:19 UTC) from {username='slave'} at 127.0.0.1:57826 I0918 20:39:19.585723 25898 pending_rounds.cc:64] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2: Trying to abort 1 pending transactions. I0918 20:39:19.590046 25898 pending_rounds.cc:70] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6425965196821626880 op_type: NO_OP noop_request { } I0918 20:39:19.590595 25898 raft_consensus.cc:2727] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted I0918 20:39:19.589877 25570 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d1c2d835755e4978a9a6dde379385a16 on 7d9b076d29dc4a24860e3f89aec6b4a0 (127.18.222.3:46373) (Table deleted at 2019-09-18 20:39:19 UTC) I0918 20:39:19.591349 25898 raft_consensus.cc:2174] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:19.591485 25899 tablet_replica.cc:273] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0: stopping tablet replica I0918 20:39:19.592260 25899 raft_consensus.cc:2147] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:19.591686 25570 catalog_manager.cc:4381] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62: Sending DeleteTablet for 3 replicas of tablet de8606c55a2e478bb968edb59bf5140f I0918 20:39:19.593626 25570 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet de8606c55a2e478bb968edb59bf5140f on c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671) (Table deleted at 2019-09-18 20:39:19 UTC) I0918 20:39:19.592654 25799 tablet_service.cc:1066] Processing DeleteTablet for tablet d1c2d835755e4978a9a6dde379385a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-18 20:39:19 UTC) from {username='slave'} at 127.0.0.1:45926 I0918 20:39:19.594822 25899 raft_consensus.cc:2174] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:19.598894 25899 ts_tablet_manager.cc:1440] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0: Deleting tablet data with delete state TABLET_DATA_DELETED I0918 20:39:19.599222 25647 tablet_service.cc:1066] Processing DeleteTablet for tablet de8606c55a2e478bb968edb59bf5140f with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-18 20:39:19 UTC) from {username='slave'} at 127.0.0.1:54632 I0918 20:39:19.601336 25570 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet de8606c55a2e478bb968edb59bf5140f on 38427f7b42664adab4dfa31963a8b7c2 (127.18.222.2:37907) (Table deleted at 2019-09-18 20:39:19 UTC) I0918 20:39:19.602882 25728 tablet_service.cc:1066] Processing DeleteTablet for tablet de8606c55a2e478bb968edb59bf5140f with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-18 20:39:19 UTC) from {username='slave'} at 127.0.0.1:57826 I0918 20:39:19.603467 25897 ts_tablet_manager.cc:1440] T ea2d24e8ebda434fa71708bf722f2054 P c8bf460a227d45cabebe5bcdb8771950: Deleting tablet data with delete state TABLET_DATA_DELETED I0918 20:39:19.604645 25898 ts_tablet_manager.cc:1440] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2: Deleting tablet data with delete state TABLET_DATA_DELETED I0918 20:39:19.603976 25570 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet de8606c55a2e478bb968edb59bf5140f on 7d9b076d29dc4a24860e3f89aec6b4a0 (127.18.222.3:46373) (Table deleted at 2019-09-18 20:39:19 UTC) I0918 20:39:19.625964 25799 tablet_service.cc:1066] Processing DeleteTablet for tablet de8606c55a2e478bb968edb59bf5140f with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-18 20:39:19 UTC) from {username='slave'} at 127.0.0.1:45926 I0918 20:39:19.631331 25899 ts_tablet_manager.cc:1453] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0 I0918 20:39:19.634318 25899 log.cc:990] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0: Deleting WAL directory at /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-2-root/wals/ea2d24e8ebda434fa71708bf722f2054 I0918 20:39:19.635192 25899 ts_tablet_manager.cc:1474] T ea2d24e8ebda434fa71708bf722f2054 P 7d9b076d29dc4a24860e3f89aec6b4a0: Deleting consensus metadata I0918 20:39:19.640496 25899 tablet_replica.cc:273] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0: stopping tablet replica I0918 20:39:19.641166 25899 raft_consensus.cc:2147] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:19.644963 25899 pending_rounds.cc:64] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0: Trying to abort 1 pending transactions. I0918 20:39:19.645210 25899 pending_rounds.cc:70] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6425965196708564992 op_type: NO_OP noop_request { } I0918 20:39:19.651305 25552 catalog_manager.cc:3501] TS 7d9b076d29dc4a24860e3f89aec6b4a0 (127.18.222.3:46373): tablet ea2d24e8ebda434fa71708bf722f2054 (table ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]) successfully deleted I0918 20:39:19.649785 25899 raft_consensus.cc:2727] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted I0918 20:39:19.683784 25899 raft_consensus.cc:2174] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:19.720890 25899 ts_tablet_manager.cc:1440] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0: Deleting tablet data with delete state TABLET_DATA_DELETED I0918 20:39:19.747087 25898 ts_tablet_manager.cc:1453] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 I0918 20:39:19.747588 25898 log.cc:990] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2: Deleting WAL directory at /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-1-root/wals/ea2d24e8ebda434fa71708bf722f2054 I0918 20:39:19.750069 25898 ts_tablet_manager.cc:1474] T ea2d24e8ebda434fa71708bf722f2054 P 38427f7b42664adab4dfa31963a8b7c2: Deleting consensus metadata I0918 20:39:19.764422 25554 catalog_manager.cc:3501] TS 38427f7b42664adab4dfa31963a8b7c2 (127.18.222.2:37907): tablet ea2d24e8ebda434fa71708bf722f2054 (table ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]) successfully deleted I0918 20:39:19.765426 25898 tablet_replica.cc:273] T d1c2d835755e4978a9a6dde379385a16 P 38427f7b42664adab4dfa31963a8b7c2: stopping tablet replica I0918 20:39:19.766060 25898 raft_consensus.cc:2147] T d1c2d835755e4978a9a6dde379385a16 P 38427f7b42664adab4dfa31963a8b7c2 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:19.766480 25898 raft_consensus.cc:2174] T d1c2d835755e4978a9a6dde379385a16 P 38427f7b42664adab4dfa31963a8b7c2 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:19.781358 25898 ts_tablet_manager.cc:1440] T d1c2d835755e4978a9a6dde379385a16 P 38427f7b42664adab4dfa31963a8b7c2: Deleting tablet data with delete state TABLET_DATA_DELETED I0918 20:39:19.784509 25869 consensus_queue.cc:984] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38427f7b42664adab4dfa31963a8b7c2" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 37907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:39:19.780684 25897 ts_tablet_manager.cc:1453] T ea2d24e8ebda434fa71708bf722f2054 P c8bf460a227d45cabebe5bcdb8771950: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0 I0918 20:39:19.787823 25897 log.cc:990] T ea2d24e8ebda434fa71708bf722f2054 P c8bf460a227d45cabebe5bcdb8771950: Deleting WAL directory at /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-0-root/wals/ea2d24e8ebda434fa71708bf722f2054 I0918 20:39:19.789327 25897 ts_tablet_manager.cc:1474] T ea2d24e8ebda434fa71708bf722f2054 P c8bf460a227d45cabebe5bcdb8771950: Deleting consensus metadata I0918 20:39:19.811859 25897 tablet_replica.cc:273] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950: stopping tablet replica I0918 20:39:19.814954 25897 raft_consensus.cc:2147] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:19.812294 25554 catalog_manager.cc:3501] TS c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671): tablet ea2d24e8ebda434fa71708bf722f2054 (table ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]) successfully deleted I0918 20:39:19.819875 25897 raft_consensus.cc:2174] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:19.821092 25571 catalog_manager.cc:4008] Got report from deleted tablet de8606c55a2e478bb968edb59bf5140f (table ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]) (Table deleted at 2019-09-18 20:39:19 UTC): Sending delete request for this tablet I0918 20:39:19.822211 25571 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet de8606c55a2e478bb968edb59bf5140f on 38427f7b42664adab4dfa31963a8b7c2 (127.18.222.2:37907) (Table deleted at 2019-09-18 20:39:19 UTC) I0918 20:39:19.822533 25897 ts_tablet_manager.cc:1440] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950: Deleting tablet data with delete state TABLET_DATA_DELETED I0918 20:39:19.823745 25728 tablet_service.cc:1066] Processing DeleteTablet for tablet de8606c55a2e478bb968edb59bf5140f with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-18 20:39:19 UTC) from {username='slave'} at 127.0.0.1:57826 I0918 20:39:19.840426 25898 ts_tablet_manager.cc:1453] T d1c2d835755e4978a9a6dde379385a16 P 38427f7b42664adab4dfa31963a8b7c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0 I0918 20:39:19.840914 25898 log.cc:990] T d1c2d835755e4978a9a6dde379385a16 P 38427f7b42664adab4dfa31963a8b7c2: Deleting WAL directory at /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-1-root/wals/d1c2d835755e4978a9a6dde379385a16 I0918 20:39:19.841928 25898 ts_tablet_manager.cc:1474] T d1c2d835755e4978a9a6dde379385a16 P 38427f7b42664adab4dfa31963a8b7c2: Deleting consensus metadata I0918 20:39:19.869628 25554 catalog_manager.cc:3501] TS 38427f7b42664adab4dfa31963a8b7c2 (127.18.222.2:37907): tablet d1c2d835755e4978a9a6dde379385a16 (table ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]) successfully deleted I0918 20:39:19.875353 25878 consensus_queue.cc:984] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8bf460a227d45cabebe5bcdb8771950" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 45671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:39:19.879525 25898 tablet_replica.cc:273] T de8606c55a2e478bb968edb59bf5140f P 38427f7b42664adab4dfa31963a8b7c2: stopping tablet replica I0918 20:39:19.882673 25568 catalog_manager.cc:4008] Got report from deleted tablet de8606c55a2e478bb968edb59bf5140f (table ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]) (Table deleted at 2019-09-18 20:39:19 UTC): Sending delete request for this tablet I0918 20:39:19.894783 25568 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet de8606c55a2e478bb968edb59bf5140f on 7d9b076d29dc4a24860e3f89aec6b4a0 (127.18.222.3:46373) (Table deleted at 2019-09-18 20:39:19 UTC) I0918 20:39:19.897578 25799 tablet_service.cc:1066] Processing DeleteTablet for tablet de8606c55a2e478bb968edb59bf5140f with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-18 20:39:19 UTC) from {username='slave'} at 127.0.0.1:45926 I0918 20:39:19.890790 25899 ts_tablet_manager.cc:1453] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 I0918 20:39:19.904724 25899 log.cc:990] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0: Deleting WAL directory at /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-2-root/wals/d1c2d835755e4978a9a6dde379385a16 I0918 20:39:19.905390 25899 ts_tablet_manager.cc:1474] T d1c2d835755e4978a9a6dde379385a16 P 7d9b076d29dc4a24860e3f89aec6b4a0: Deleting consensus metadata I0918 20:39:19.887995 25898 raft_consensus.cc:2147] T de8606c55a2e478bb968edb59bf5140f P 38427f7b42664adab4dfa31963a8b7c2 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:19.911265 25568 catalog_manager.cc:4008] Got report from deleted tablet de8606c55a2e478bb968edb59bf5140f (table ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]) (Table deleted at 2019-09-18 20:39:19 UTC): Sending delete request for this tablet I0918 20:39:19.913041 25899 tablet_replica.cc:273] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0: stopping tablet replica I0918 20:39:19.913713 25899 raft_consensus.cc:2147] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:19.914599 25899 raft_consensus.cc:2174] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:19.915450 25568 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet de8606c55a2e478bb968edb59bf5140f on 38427f7b42664adab4dfa31963a8b7c2 (127.18.222.2:37907) (Table deleted at 2019-09-18 20:39:19 UTC) I0918 20:39:19.917248 25898 raft_consensus.cc:2174] T de8606c55a2e478bb968edb59bf5140f P 38427f7b42664adab4dfa31963a8b7c2 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:19.913628 25552 catalog_manager.cc:3501] TS 7d9b076d29dc4a24860e3f89aec6b4a0 (127.18.222.3:46373): tablet d1c2d835755e4978a9a6dde379385a16 (table ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]) successfully deleted I0918 20:39:19.920163 25728 tablet_service.cc:1066] Processing DeleteTablet for tablet de8606c55a2e478bb968edb59bf5140f with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-18 20:39:19 UTC) from {username='slave'} at 127.0.0.1:57826 I0918 20:39:19.936460 25898 ts_tablet_manager.cc:1440] T de8606c55a2e478bb968edb59bf5140f P 38427f7b42664adab4dfa31963a8b7c2: Deleting tablet data with delete state TABLET_DATA_DELETED I0918 20:39:19.958288 25899 ts_tablet_manager.cc:1440] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0: Deleting tablet data with delete state TABLET_DATA_DELETED I0918 20:39:19.976989 25570 catalog_manager.cc:4008] Got report from deleted tablet de8606c55a2e478bb968edb59bf5140f (table ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]) (Table deleted at 2019-09-18 20:39:19 UTC): Sending delete request for this tablet I0918 20:39:19.978288 25570 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet de8606c55a2e478bb968edb59bf5140f on c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671) (Table deleted at 2019-09-18 20:39:19 UTC) I0918 20:39:19.981035 25899 ts_tablet_manager.cc:1453] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 I0918 20:39:19.981560 25899 log.cc:990] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0: Deleting WAL directory at /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-2-root/wals/de8606c55a2e478bb968edb59bf5140f I0918 20:39:19.982473 25899 ts_tablet_manager.cc:1474] T de8606c55a2e478bb968edb59bf5140f P 7d9b076d29dc4a24860e3f89aec6b4a0: Deleting consensus metadata I0918 20:39:19.986387 25647 tablet_service.cc:1066] Processing DeleteTablet for tablet de8606c55a2e478bb968edb59bf5140f with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-18 20:39:19 UTC) from {username='slave'} at 127.0.0.1:54632 I0918 20:39:19.985464 25570 catalog_manager.cc:4008] Got report from deleted tablet de8606c55a2e478bb968edb59bf5140f (table ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]) (Table deleted at 2019-09-18 20:39:19 UTC): Sending delete request for this tablet I0918 20:39:19.990471 25570 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet de8606c55a2e478bb968edb59bf5140f on c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671) (Table deleted at 2019-09-18 20:39:19 UTC) I0918 20:39:19.992065 25647 tablet_service.cc:1066] Processing DeleteTablet for tablet de8606c55a2e478bb968edb59bf5140f with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-18 20:39:19 UTC) from {username='slave'} at 127.0.0.1:54632 I0918 20:39:19.995370 25552 catalog_manager.cc:3501] TS 7d9b076d29dc4a24860e3f89aec6b4a0 (127.18.222.3:46373): tablet de8606c55a2e478bb968edb59bf5140f (table ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]) successfully deleted W0918 20:39:19.996366 25552 catalog_manager.cc:3481] TS 7d9b076d29dc4a24860e3f89aec6b4a0 (127.18.222.3:46373): delete failed for tablet de8606c55a2e478bb968edb59bf5140f because the tablet was not found. No further retry: Not found: Tablet not found: de8606c55a2e478bb968edb59bf5140f I0918 20:39:20.010088 25898 ts_tablet_manager.cc:1453] T de8606c55a2e478bb968edb59bf5140f P 38427f7b42664adab4dfa31963a8b7c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 I0918 20:39:20.014266 25898 log.cc:990] T de8606c55a2e478bb968edb59bf5140f P 38427f7b42664adab4dfa31963a8b7c2: Deleting WAL directory at /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-1-root/wals/de8606c55a2e478bb968edb59bf5140f I0918 20:39:20.015450 25898 ts_tablet_manager.cc:1474] T de8606c55a2e478bb968edb59bf5140f P 38427f7b42664adab4dfa31963a8b7c2: Deleting consensus metadata I0918 20:39:20.018723 25897 ts_tablet_manager.cc:1453] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0 I0918 20:39:20.019168 25897 log.cc:990] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950: Deleting WAL directory at /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-0-root/wals/d1c2d835755e4978a9a6dde379385a16 I0918 20:39:20.020011 25897 ts_tablet_manager.cc:1474] T d1c2d835755e4978a9a6dde379385a16 P c8bf460a227d45cabebe5bcdb8771950: Deleting consensus metadata I0918 20:39:20.020815 25554 catalog_manager.cc:3501] TS 38427f7b42664adab4dfa31963a8b7c2 (127.18.222.2:37907): tablet de8606c55a2e478bb968edb59bf5140f (table ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]) successfully deleted I0918 20:39:20.023018 25554 catalog_manager.cc:3501] TS c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671): tablet d1c2d835755e4978a9a6dde379385a16 (table ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]) successfully deleted W0918 20:39:20.026310 25554 catalog_manager.cc:3481] TS 38427f7b42664adab4dfa31963a8b7c2 (127.18.222.2:37907): delete failed for tablet de8606c55a2e478bb968edb59bf5140f because the tablet was not found. No further retry: Not found: Tablet not found: de8606c55a2e478bb968edb59bf5140f W0918 20:39:20.027947 25554 catalog_manager.cc:3481] TS 38427f7b42664adab4dfa31963a8b7c2 (127.18.222.2:37907): delete failed for tablet de8606c55a2e478bb968edb59bf5140f because the tablet was not found. No further retry: Not found: Tablet not found: de8606c55a2e478bb968edb59bf5140f I0918 20:39:20.029052 25897 tablet_replica.cc:273] T de8606c55a2e478bb968edb59bf5140f P c8bf460a227d45cabebe5bcdb8771950: stopping tablet replica I0918 20:39:20.029774 25897 raft_consensus.cc:2147] T de8606c55a2e478bb968edb59bf5140f P c8bf460a227d45cabebe5bcdb8771950 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:20.030303 25897 raft_consensus.cc:2174] T de8606c55a2e478bb968edb59bf5140f P c8bf460a227d45cabebe5bcdb8771950 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:20.034535 25897 ts_tablet_manager.cc:1440] T de8606c55a2e478bb968edb59bf5140f P c8bf460a227d45cabebe5bcdb8771950: Deleting tablet data with delete state TABLET_DATA_DELETED I0918 20:39:20.090879 25897 ts_tablet_manager.cc:1453] T de8606c55a2e478bb968edb59bf5140f P c8bf460a227d45cabebe5bcdb8771950: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 I0918 20:39:20.091279 25897 log.cc:990] T de8606c55a2e478bb968edb59bf5140f P c8bf460a227d45cabebe5bcdb8771950: Deleting WAL directory at /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1568839089958813-19320/minicluster-data/ts-0-root/wals/de8606c55a2e478bb968edb59bf5140f I0918 20:39:20.092411 25897 ts_tablet_manager.cc:1474] T de8606c55a2e478bb968edb59bf5140f P c8bf460a227d45cabebe5bcdb8771950: Deleting consensus metadata I0918 20:39:20.098321 25554 catalog_manager.cc:3501] TS c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671): tablet de8606c55a2e478bb968edb59bf5140f (table ksck-test-table [id=714964d98e494182ad5e27022fe25fcf]) successfully deleted W0918 20:39:20.099010 25554 catalog_manager.cc:3481] TS c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671): delete failed for tablet de8606c55a2e478bb968edb59bf5140f because the tablet was not found. No further retry: Not found: Tablet not found: de8606c55a2e478bb968edb59bf5140f W0918 20:39:20.099654 25554 catalog_manager.cc:3481] TS c8bf460a227d45cabebe5bcdb8771950 (127.18.222.1:45671): delete failed for tablet de8606c55a2e478bb968edb59bf5140f because the tablet was not found. No further retry: Not found: Tablet not found: de8606c55a2e478bb968edb59bf5140f I0918 20:39:20.359112 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:20.429292 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:20.443343 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:20.443552 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:20.443974 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:20.457408 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:20.492368 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:20.512495 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:20.512856 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:20.513411 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:20.525717 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:20.560360 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:20.576310 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:20.576508 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:20.576894 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:20.588063 19320 master.cc:247] Master@127.18.222.62:36069 shutting down... I0918 20:39:20.604604 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:20.605008 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:20.605254 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P b8424e7d64a648f3952fd83c45ad7fc4: stopping tablet replica I0918 20:39:20.623760 19320 master.cc:258] Master@127.18.222.62:36069 shutdown complete. I0918 20:39:20.633833 19320 master.cc:247] Master@127.18.222.61:38137 shutting down... I0918 20:39:20.651280 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:20.651655 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:20.651968 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 5f649e2399b743d9a53a42952539e941: stopping tablet replica I0918 20:39:20.669979 19320 master.cc:258] Master@127.18.222.61:38137 shutdown complete. I0918 20:39:20.681376 19320 master.cc:247] Master@127.18.222.60:41641 shutting down... I0918 20:39:20.701153 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:20.701943 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:20.702204 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P d38eb92d8be54a50953dc5aed9dc8c62: stopping tablet replica I0918 20:39:20.721235 19320 master.cc:258] Master@127.18.222.60:41641 shutdown complete. [ OK ] RemoteKsckTest.TestFilterOnNoTableCluster (3311 ms) [ RUN ] RemoteKsckTest.TestNonMatchingTableFilter I0918 20:39:20.754408 19320 test_util.cc:212] Using random seed: -723324722 I0918 20:39:20.755146 19320 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.18.222.62:34959,127.18.222.61:43721,127.18.222.60:33177 I0918 20:39:20.767192 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 157592us I0918 20:39:20.767582 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:20.767925 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:20.768141 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:20.768267 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:20.769268 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/master-0-root/instance: uuid: "b27de89bfdca4ef4a3baf93a01c8d303" format_stamp: "Formatted at 2019-09-18 20:39:20 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:20.777644 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.008s sys 0.001s I0918 20:39:20.779069 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:20.785948 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:20.786126 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/master-0-root uuid: "b27de89bfdca4ef4a3baf93a01c8d303" format_stamp: "Formatted at 2019-09-18 20:39:20 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:20.786417 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:20.863281 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:34959 I0918 20:39:20.865561 19320 webserver.cc:374] Webserver started at http://127.18.222.62:38271/ using document root and password file I0918 20:39:20.867983 25992 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:20.876080 25992 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:20.886466 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 157592us I0918 20:39:20.887181 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:20.887447 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:20.889816 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:20.890063 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:20.891530 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/master-1-root/instance: uuid: "b1328916d51a41baa01800e39f6e6ad2" format_stamp: "Formatted at 2019-09-18 20:39:20 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:20.900817 25992 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:20.903216 25992 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:43721: Network error: Client connection negotiation failed: client connection to 127.18.222.61:43721: connect: Connection refused (error 111) I0918 20:39:20.905355 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.013s user 0.004s sys 0.009s I0918 20:39:20.907125 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:20.915303 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:39:20.915516 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/master-1-root uuid: "b1328916d51a41baa01800e39f6e6ad2" format_stamp: "Formatted at 2019-09-18 20:39:20 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:20.915848 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:20.958884 25992 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } attempt: 1 W0918 20:39:20.961441 25992 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:43721: Network error: Client connection negotiation failed: client connection to 127.18.222.61:43721: connect: Connection refused (error 111) I0918 20:39:20.971421 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:43721 I0918 20:39:20.973500 19320 webserver.cc:374] Webserver started at http://127.18.222.61:43799/ using document root and password file I0918 20:39:20.975528 26049 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:20.982744 26049 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:20.992189 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 157592us I0918 20:39:20.992789 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:20.993019 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:20.993327 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:20.993518 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:20.994940 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/master-2-root/instance: uuid: "3026503f76814232b3a196172349ca5c" format_stamp: "Formatted at 2019-09-18 20:39:20 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:20.997090 26049 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:21.011792 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.016s user 0.017s sys 0.000s I0918 20:39:21.012233 26049 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:21.013972 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 W0918 20:39:21.014369 26049 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:33177: Network error: Client connection negotiation failed: client connection to 127.18.222.60:33177: connect: Connection refused (error 111) I0918 20:39:21.022691 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:21.022891 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/master-2-root uuid: "3026503f76814232b3a196172349ca5c" format_stamp: "Formatted at 2019-09-18 20:39:20 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:21.023133 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:21.040948 25992 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } attempt: 2 I0918 20:39:21.054625 25992 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:21.057044 25992 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:33177: Network error: Client connection negotiation failed: client connection to 127.18.222.60:33177: connect: Connection refused (error 111) I0918 20:39:21.075894 26049 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } attempt: 1 W0918 20:39:21.078399 26049 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:33177: Network error: Client connection negotiation failed: client connection to 127.18.222.60:33177: connect: Connection refused (error 111) I0918 20:39:21.080790 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:33177 I0918 20:39:21.082861 19320 webserver.cc:374] Webserver started at http://127.18.222.60:40771/ using document root and password file I0918 20:39:21.083927 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0918 20:39:21.084856 26106 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:21.091032 26106 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:21.103469 26106 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:21.114353 26106 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:21.114464 25992 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } attempt: 1 I0918 20:39:21.146459 26106 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c: Bootstrap starting. I0918 20:39:21.146459 25992 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303: Bootstrap starting. I0918 20:39:21.151224 25992 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303: No blocks or log segments found. Creating new log. I0918 20:39:21.151224 26106 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c: No blocks or log segments found. Creating new log. I0918 20:39:21.154844 26049 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } attempt: 2 I0918 20:39:21.156263 25992 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303: No bootstrap required, opened a new log I0918 20:39:21.157200 26106 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c: No bootstrap required, opened a new log I0918 20:39:21.159209 25992 raft_consensus.cc:340] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } I0918 20:39:21.159891 25992 raft_consensus.cc:366] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:21.159482 26106 raft_consensus.cc:340] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } I0918 20:39:21.160151 25992 raft_consensus.cc:705] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b27de89bfdca4ef4a3baf93a01c8d303, State: Initialized, Role: FOLLOWER I0918 20:39:21.160306 26106 raft_consensus.cc:366] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:21.160579 26106 raft_consensus.cc:705] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3026503f76814232b3a196172349ca5c, State: Initialized, Role: FOLLOWER I0918 20:39:21.161228 26106 consensus_queue.cc:229] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } I0918 20:39:21.160907 25992 consensus_queue.cc:229] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } I0918 20:39:21.163321 26106 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } } I0918 20:39:21.164425 26106 sys_catalog.cc:343] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:21.164211 25992 sys_catalog.cc:340] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } } I0918 20:39:21.164497 26113 sys_catalog.cc:340] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } } I0918 20:39:21.165395 26112 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } } I0918 20:39:21.165789 26106 sys_catalog.cc:428] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:21.166424 26112 sys_catalog.cc:343] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:21.165143 25992 sys_catalog.cc:343] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:21.166124 26113 sys_catalog.cc:343] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:21.167948 25992 sys_catalog.cc:428] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:21.169908 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 W0918 20:39:21.172250 26115 catalog_manager.cc:1155] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:21.172260 26114 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:21.172772 26115 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0918 20:39:21.172889 26114 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:21.176271 26049 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2: Bootstrap starting. I0918 20:39:21.179302 26049 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2: No blocks or log segments found. Creating new log. I0918 20:39:21.182796 26049 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2: No bootstrap required, opened a new log I0918 20:39:21.184504 26049 raft_consensus.cc:340] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } I0918 20:39:21.184995 26049 raft_consensus.cc:366] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:21.185199 26049 raft_consensus.cc:705] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1328916d51a41baa01800e39f6e6ad2, State: Initialized, Role: FOLLOWER I0918 20:39:21.185714 26049 consensus_queue.cc:229] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } I0918 20:39:21.187381 26049 sys_catalog.cc:340] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } } I0918 20:39:21.188211 26049 sys_catalog.cc:343] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:21.187731 26117 sys_catalog.cc:340] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } } I0918 20:39:21.188594 26117 sys_catalog.cc:343] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:21.189376 26049 sys_catalog.cc:428] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:21.190560 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 W0918 20:39:21.192698 26118 catalog_manager.cc:1155] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:21.192977 26118 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:21.206347 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 158092us I0918 20:39:21.206791 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:21.206967 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:21.207190 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:21.207319 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:21.208678 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-0-root/instance: uuid: "184d7d5558a34378a172945529659ee6" format_stamp: "Formatted at 2019-09-18 20:39:21 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:21.219028 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.010s sys 0.000s I0918 20:39:21.220837 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:21.229178 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:21.229434 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-0-root uuid: "184d7d5558a34378a172945529659ee6" format_stamp: "Formatted at 2019-09-18 20:39:21 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:21.229732 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:21.261471 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:21.261638 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:21.309340 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:46755 I0918 20:39:21.313696 19320 webserver.cc:374] Webserver started at http://127.18.222.1:36247/ using document root and password file I0918 20:39:21.341090 26183 heartbeater.cc:322] Connected to a master server at 127.18.222.60:33177 I0918 20:39:21.341642 26183 heartbeater.cc:406] Registering TS with master... I0918 20:39:21.342545 26183 heartbeater.cc:455] Master 127.18.222.60:33177 requested a full tablet report, sending... I0918 20:39:21.342597 26184 heartbeater.cc:322] Connected to a master server at 127.18.222.61:43721 I0918 20:39:21.343016 26184 heartbeater.cc:406] Registering TS with master... I0918 20:39:21.343551 26184 heartbeater.cc:455] Master 127.18.222.61:43721 requested a full tablet report, sending... I0918 20:39:21.345702 26073 ts_manager.cc:186] Registered new tserver with Master: 184d7d5558a34378a172945529659ee6 (127.18.222.1:46755) I0918 20:39:21.346320 26016 ts_manager.cc:186] Registered new tserver with Master: 184d7d5558a34378a172945529659ee6 (127.18.222.1:46755) I0918 20:39:21.348829 26187 heartbeater.cc:322] Connected to a master server at 127.18.222.62:34959 I0918 20:39:21.349105 26187 heartbeater.cc:406] Registering TS with master... I0918 20:39:21.349615 26187 heartbeater.cc:455] Master 127.18.222.62:34959 requested a full tablet report, sending... I0918 20:39:21.351812 25958 ts_manager.cc:186] Registered new tserver with Master: 184d7d5558a34378a172945529659ee6 (127.18.222.1:46755) I0918 20:39:21.354902 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 158092us I0918 20:39:21.355304 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:21.355490 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:21.355743 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:21.355887 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:21.356989 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-1-root/instance: uuid: "bd6134f31529493282c7cf50de21bfc3" format_stamp: "Formatted at 2019-09-18 20:39:21 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:21.368486 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.012s sys 0.000s I0918 20:39:21.370146 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:21.380128 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0918 20:39:21.380362 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-1-root uuid: "bd6134f31529493282c7cf50de21bfc3" format_stamp: "Formatted at 2019-09-18 20:39:21 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:21.380616 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:21.408813 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:21.409024 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:21.438627 26117 raft_consensus.cc:472] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:21.439404 26117 raft_consensus.cc:494] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } I0918 20:39:21.441324 26117 leader_election.cc:296] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b27de89bfdca4ef4a3baf93a01c8d303 (127.18.222.62:34959), 3026503f76814232b3a196172349ca5c (127.18.222.60:33177) I0918 20:39:21.442577 25969 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b1328916d51a41baa01800e39f6e6ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" is_pre_election: true I0918 20:39:21.442836 26083 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b1328916d51a41baa01800e39f6e6ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3026503f76814232b3a196172349ca5c" is_pre_election: true I0918 20:39:21.444181 26083 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1328916d51a41baa01800e39f6e6ad2 in term 0. I0918 20:39:21.445844 25999 leader_election.cc:310] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3026503f76814232b3a196172349ca5c, b1328916d51a41baa01800e39f6e6ad2; no voters: I0918 20:39:21.447050 25969 raft_consensus.cc:2350] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1328916d51a41baa01800e39f6e6ad2 in term 0. I0918 20:39:21.448846 26117 raft_consensus.cc:2644] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:21.449224 26117 raft_consensus.cc:472] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:21.449597 26117 raft_consensus.cc:2890] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:21.459430 26117 raft_consensus.cc:494] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } I0918 20:39:21.462270 26117 leader_election.cc:296] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [CANDIDATE]: Term 1 election: Requested vote from peers b27de89bfdca4ef4a3baf93a01c8d303 (127.18.222.62:34959), 3026503f76814232b3a196172349ca5c (127.18.222.60:33177) I0918 20:39:21.463187 25969 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b1328916d51a41baa01800e39f6e6ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" I0918 20:39:21.463379 26083 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b1328916d51a41baa01800e39f6e6ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3026503f76814232b3a196172349ca5c" I0918 20:39:21.463951 25969 raft_consensus.cc:2890] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:21.464083 26083 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:21.470050 25969 raft_consensus.cc:2350] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1328916d51a41baa01800e39f6e6ad2 in term 1. I0918 20:39:21.471248 25997 leader_election.cc:310] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1328916d51a41baa01800e39f6e6ad2, b27de89bfdca4ef4a3baf93a01c8d303; no voters: I0918 20:39:21.472048 26117 raft_consensus.cc:2644] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:21.474596 26083 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1328916d51a41baa01800e39f6e6ad2 in term 1. I0918 20:39:21.479300 26117 raft_consensus.cc:667] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [term 1 LEADER]: Becoming Leader. State: Replica: b1328916d51a41baa01800e39f6e6ad2, State: Running, Role: LEADER I0918 20:39:21.480139 26117 consensus_queue.cc:206] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } I0918 20:39:21.485822 26238 sys_catalog.cc:340] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1328916d51a41baa01800e39f6e6ad2. Latest consensus state: current_term: 1 leader_uuid: "b1328916d51a41baa01800e39f6e6ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } } I0918 20:39:21.486410 26238 sys_catalog.cc:343] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [sys.catalog]: This master's current role is: LEADER I0918 20:39:21.488750 26244 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:39:21.494309 26244 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:39:21.507133 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.2:34567 I0918 20:39:21.510185 19320 webserver.cc:374] Webserver started at http://127.18.222.2:34813/ using document root and password file I0918 20:39:21.549676 26083 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c [term 1 FOLLOWER]: Refusing update from remote peer b1328916d51a41baa01800e39f6e6ad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:21.551543 26238 consensus_queue.cc:984] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:21.557869 25969 raft_consensus.cc:1184] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303 [term 1 FOLLOWER]: Refusing update from remote peer b1328916d51a41baa01800e39f6e6ad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:21.559667 26238 consensus_queue.cc:984] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:21.570667 26112 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1328916d51a41baa01800e39f6e6ad2. Latest consensus state: current_term: 1 leader_uuid: "b1328916d51a41baa01800e39f6e6ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } } I0918 20:39:21.571488 26112 sys_catalog.cc:343] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:21.575037 26113 sys_catalog.cc:340] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1328916d51a41baa01800e39f6e6ad2. Latest consensus state: current_term: 1 leader_uuid: "b1328916d51a41baa01800e39f6e6ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } } I0918 20:39:21.576609 26113 sys_catalog.cc:343] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:21.577595 26257 heartbeater.cc:322] Connected to a master server at 127.18.222.60:33177 I0918 20:39:21.577980 26257 heartbeater.cc:406] Registering TS with master... I0918 20:39:21.586210 26257 heartbeater.cc:455] Master 127.18.222.60:33177 requested a full tablet report, sending... I0918 20:39:21.592856 26260 heartbeater.cc:322] Connected to a master server at 127.18.222.62:34959 I0918 20:39:21.593355 26260 heartbeater.cc:406] Registering TS with master... I0918 20:39:21.594135 26260 heartbeater.cc:455] Master 127.18.222.62:34959 requested a full tablet report, sending... I0918 20:39:21.595398 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 158092us I0918 20:39:21.596165 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:21.596904 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:21.597142 26073 ts_manager.cc:186] Registered new tserver with Master: bd6134f31529493282c7cf50de21bfc3 (127.18.222.2:34567) I0918 20:39:21.597173 25958 ts_manager.cc:186] Registered new tserver with Master: bd6134f31529493282c7cf50de21bfc3 (127.18.222.2:34567) I0918 20:39:21.597551 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:21.598225 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:21.602509 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-2-root/instance: uuid: "4462c4818a1647ef9513032e7213c193" format_stamp: "Formatted at 2019-09-18 20:39:21 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:21.601126 26117 sys_catalog.cc:340] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b1328916d51a41baa01800e39f6e6ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } } I0918 20:39:21.604084 26117 sys_catalog.cc:343] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [sys.catalog]: This master's current role is: LEADER I0918 20:39:21.604418 26258 heartbeater.cc:322] Connected to a master server at 127.18.222.61:43721 I0918 20:39:21.604827 26258 heartbeater.cc:406] Registering TS with master... I0918 20:39:21.605443 26258 heartbeater.cc:455] Master 127.18.222.61:43721 requested a full tablet report, sending... I0918 20:39:21.607972 26016 ts_manager.cc:186] Registered new tserver with Master: bd6134f31529493282c7cf50de21bfc3 (127.18.222.2:34567) I0918 20:39:21.610620 26269 sys_catalog.cc:340] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b1328916d51a41baa01800e39f6e6ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } } I0918 20:39:21.614089 26269 sys_catalog.cc:343] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [sys.catalog]: This master's current role is: LEADER I0918 20:39:21.616410 26112 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b1328916d51a41baa01800e39f6e6ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } } I0918 20:39:21.617166 26112 sys_catalog.cc:343] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:21.618718 26244 catalog_manager.cc:981] Generated new certificate authority record I0918 20:39:21.620168 26113 sys_catalog.cc:340] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b1328916d51a41baa01800e39f6e6ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27de89bfdca4ef4a3baf93a01c8d303" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 34959 } } peers { permanent_uuid: "b1328916d51a41baa01800e39f6e6ad2" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43721 } } peers { permanent_uuid: "3026503f76814232b3a196172349ca5c" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 33177 } } } I0918 20:39:21.620992 26113 sys_catalog.cc:343] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:21.624065 26244 catalog_manager.cc:1106] Loading token signing keys... I0918 20:39:21.632917 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.030s user 0.014s sys 0.007s I0918 20:39:21.635116 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:21.651392 19320 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s I0918 20:39:21.651677 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-2-root uuid: "4462c4818a1647ef9513032e7213c193" format_stamp: "Formatted at 2019-09-18 20:39:21 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:21.652089 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:21.656548 26244 catalog_manager.cc:4446] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2: Generated new TSK 0 I0918 20:39:21.663758 26244 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:39:21.678745 26016 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34004 I0918 20:39:21.680073 26015 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34010 I0918 20:39:21.695314 26258 heartbeater.cc:447] Master 127.18.222.61:43721 was elected leader, sending a full tablet report... I0918 20:39:21.696151 26184 heartbeater.cc:447] Master 127.18.222.61:43721 was elected leader, sending a full tablet report... I0918 20:39:21.719674 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:21.720021 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:21.782682 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.3:39999 I0918 20:39:21.785718 19320 webserver.cc:374] Webserver started at http://127.18.222.3:46071/ using document root and password file I0918 20:39:21.833537 26337 heartbeater.cc:322] Connected to a master server at 127.18.222.60:33177 I0918 20:39:21.833863 26337 heartbeater.cc:406] Registering TS with master... I0918 20:39:21.834431 26337 heartbeater.cc:455] Master 127.18.222.60:33177 requested a full tablet report, sending... I0918 20:39:21.837127 26073 ts_manager.cc:186] Registered new tserver with Master: 4462c4818a1647ef9513032e7213c193 (127.18.222.3:39999) I0918 20:39:21.840556 26338 heartbeater.cc:322] Connected to a master server at 127.18.222.61:43721 I0918 20:39:21.840859 26338 heartbeater.cc:406] Registering TS with master... I0918 20:39:21.841428 26338 heartbeater.cc:455] Master 127.18.222.61:43721 requested a full tablet report, sending... I0918 20:39:21.843775 26015 ts_manager.cc:186] Registered new tserver with Master: 4462c4818a1647ef9513032e7213c193 (127.18.222.3:39999) I0918 20:39:21.844954 26339 heartbeater.cc:322] Connected to a master server at 127.18.222.62:34959 I0918 20:39:21.845444 26339 heartbeater.cc:406] Registering TS with master... I0918 20:39:21.846094 26339 heartbeater.cc:455] Master 127.18.222.62:34959 requested a full tablet report, sending... I0918 20:39:21.846285 26015 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34016 I0918 20:39:21.848892 25957 ts_manager.cc:186] Registered new tserver with Master: 4462c4818a1647ef9513032e7213c193 (127.18.222.3:39999) I0918 20:39:21.848913 19320 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.042536014s I0918 20:39:21.860182 26338 heartbeater.cc:447] Master 127.18.222.61:43721 was elected leader, sending a full tablet report... I0918 20:39:21.924387 26015 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34022: name: "ksck-test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\n\000\000\000\004\001\024\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:39:21.925886 26015 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:39:22.065575 26304 tablet_service.cc:1022] Processing CreateTablet for tablet 3cf1523b639146c29bbf465d50405209 (table=ksck-test-table [id=25a2e165edef4726b9f859fc6c4922bb]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:22.066651 26304 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 3cf1523b639146c29bbf465d50405209. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:22.068461 26303 tablet_service.cc:1022] Processing CreateTablet for tablet fe1a99b16020484a88a8f7888095479a (table=ksck-test-table [id=25a2e165edef4726b9f859fc6c4922bb]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:22.069519 26303 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet fe1a99b16020484a88a8f7888095479a. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:22.072543 26302 tablet_service.cc:1022] Processing CreateTablet for tablet 891bd0a07cc944bfb77b70deae4de632 (table=ksck-test-table [id=25a2e165edef4726b9f859fc6c4922bb]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:22.073638 26302 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 891bd0a07cc944bfb77b70deae4de632. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:22.098830 26221 tablet_service.cc:1022] Processing CreateTablet for tablet 3cf1523b639146c29bbf465d50405209 (table=ksck-test-table [id=25a2e165edef4726b9f859fc6c4922bb]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:22.104542 26221 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 3cf1523b639146c29bbf465d50405209. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:22.104822 26220 tablet_service.cc:1022] Processing CreateTablet for tablet fe1a99b16020484a88a8f7888095479a (table=ksck-test-table [id=25a2e165edef4726b9f859fc6c4922bb]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:22.105877 26220 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet fe1a99b16020484a88a8f7888095479a. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:22.105983 26150 tablet_service.cc:1022] Processing CreateTablet for tablet 3cf1523b639146c29bbf465d50405209 (table=ksck-test-table [id=25a2e165edef4726b9f859fc6c4922bb]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:22.106806 26150 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 3cf1523b639146c29bbf465d50405209. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:22.108427 26148 tablet_service.cc:1022] Processing CreateTablet for tablet 891bd0a07cc944bfb77b70deae4de632 (table=ksck-test-table [id=25a2e165edef4726b9f859fc6c4922bb]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:22.109386 26148 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 891bd0a07cc944bfb77b70deae4de632. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:22.110270 26149 tablet_service.cc:1022] Processing CreateTablet for tablet fe1a99b16020484a88a8f7888095479a (table=ksck-test-table [id=25a2e165edef4726b9f859fc6c4922bb]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:22.111142 26149 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet fe1a99b16020484a88a8f7888095479a. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:22.112324 26219 tablet_service.cc:1022] Processing CreateTablet for tablet 891bd0a07cc944bfb77b70deae4de632 (table=ksck-test-table [id=25a2e165edef4726b9f859fc6c4922bb]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:22.113379 26219 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 891bd0a07cc944bfb77b70deae4de632. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:22.206954 26358 tablet_bootstrap.cc:439] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193: Bootstrap starting. I0918 20:39:22.216033 26358 tablet_bootstrap.cc:590] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193: No blocks or log segments found. Creating new log. I0918 20:39:22.220398 26359 tablet_bootstrap.cc:439] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6: Bootstrap starting. I0918 20:39:22.228643 26114 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c: acquiring CA information for follower catalog manager: success I0918 20:39:22.237409 26114 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:22.238054 26359 tablet_bootstrap.cc:590] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6: No blocks or log segments found. Creating new log. I0918 20:39:22.243443 26360 tablet_bootstrap.cc:439] T 3cf1523b639146c29bbf465d50405209 P bd6134f31529493282c7cf50de21bfc3: Bootstrap starting. I0918 20:39:22.248011 26358 tablet_bootstrap.cc:439] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193: No bootstrap required, opened a new log I0918 20:39:22.261396 26358 ts_tablet_manager.cc:1085] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193: Time spent bootstrapping tablet: real 0.055s user 0.009s sys 0.000s I0918 20:39:22.247474 26115 catalog_manager.cc:1153] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303: acquiring CA information for follower catalog manager: success I0918 20:39:22.263490 26358 raft_consensus.cc:340] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.265017 26358 raft_consensus.cc:366] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:22.265362 26358 raft_consensus.cc:705] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4462c4818a1647ef9513032e7213c193, State: Initialized, Role: FOLLOWER I0918 20:39:22.265630 26360 tablet_bootstrap.cc:590] T 3cf1523b639146c29bbf465d50405209 P bd6134f31529493282c7cf50de21bfc3: No blocks or log segments found. Creating new log. I0918 20:39:22.266201 26358 consensus_queue.cc:229] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.268940 26358 ts_tablet_manager.cc:1113] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s I0918 20:39:22.269692 26115 catalog_manager.cc:1181] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:22.269783 26358 tablet_bootstrap.cc:439] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193: Bootstrap starting. I0918 20:39:22.280292 26358 tablet_bootstrap.cc:590] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193: No blocks or log segments found. Creating new log. I0918 20:39:22.283200 26359 tablet_bootstrap.cc:439] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6: No bootstrap required, opened a new log I0918 20:39:22.283754 26359 ts_tablet_manager.cc:1085] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6: Time spent bootstrapping tablet: real 0.064s user 0.005s sys 0.012s I0918 20:39:22.286145 26359 raft_consensus.cc:340] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.286936 26359 raft_consensus.cc:366] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:22.287219 26359 raft_consensus.cc:705] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184d7d5558a34378a172945529659ee6, State: Initialized, Role: FOLLOWER I0918 20:39:22.288079 26359 consensus_queue.cc:229] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.297521 26358 tablet_bootstrap.cc:439] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193: No bootstrap required, opened a new log I0918 20:39:22.298265 26358 ts_tablet_manager.cc:1085] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.000s I0918 20:39:22.298710 26360 tablet_bootstrap.cc:439] T 3cf1523b639146c29bbf465d50405209 P bd6134f31529493282c7cf50de21bfc3: No bootstrap required, opened a new log I0918 20:39:22.299341 26360 ts_tablet_manager.cc:1085] T 3cf1523b639146c29bbf465d50405209 P bd6134f31529493282c7cf50de21bfc3: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.009s I0918 20:39:22.316476 26359 ts_tablet_manager.cc:1113] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6: Time spent starting tablet: real 0.025s user 0.006s sys 0.011s I0918 20:39:22.317728 26359 tablet_bootstrap.cc:439] T 3cf1523b639146c29bbf465d50405209 P 184d7d5558a34378a172945529659ee6: Bootstrap starting. I0918 20:39:22.318785 26360 raft_consensus.cc:340] T 3cf1523b639146c29bbf465d50405209 P bd6134f31529493282c7cf50de21bfc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } I0918 20:39:22.319999 26360 raft_consensus.cc:366] T 3cf1523b639146c29bbf465d50405209 P bd6134f31529493282c7cf50de21bfc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:22.320279 26360 raft_consensus.cc:705] T 3cf1523b639146c29bbf465d50405209 P bd6134f31529493282c7cf50de21bfc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6134f31529493282c7cf50de21bfc3, State: Initialized, Role: FOLLOWER I0918 20:39:22.321130 26360 consensus_queue.cc:229] T 3cf1523b639146c29bbf465d50405209 P bd6134f31529493282c7cf50de21bfc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } I0918 20:39:22.324956 26359 tablet_bootstrap.cc:590] T 3cf1523b639146c29bbf465d50405209 P 184d7d5558a34378a172945529659ee6: No blocks or log segments found. Creating new log. I0918 20:39:22.328986 26366 raft_consensus.cc:472] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:22.330737 26366 raft_consensus.cc:494] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.332932 26358 raft_consensus.cc:340] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } I0918 20:39:22.333926 26358 raft_consensus.cc:366] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:22.334199 26358 raft_consensus.cc:705] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4462c4818a1647ef9513032e7213c193, State: Initialized, Role: FOLLOWER I0918 20:39:22.335033 26358 consensus_queue.cc:229] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } I0918 20:39:22.341689 26363 raft_consensus.cc:472] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:22.342178 26363 raft_consensus.cc:494] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.354506 26358 ts_tablet_manager.cc:1113] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193: Time spent starting tablet: real 0.056s user 0.006s sys 0.000s I0918 20:39:22.348104 26366 leader_election.cc:296] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6134f31529493282c7cf50de21bfc3 (127.18.222.2:34567), 4462c4818a1647ef9513032e7213c193 (127.18.222.3:39999) I0918 20:39:22.365520 26360 ts_tablet_manager.cc:1113] T 3cf1523b639146c29bbf465d50405209 P bd6134f31529493282c7cf50de21bfc3: Time spent starting tablet: real 0.066s user 0.005s sys 0.003s I0918 20:39:22.365748 26363 leader_election.cc:296] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184d7d5558a34378a172945529659ee6 (127.18.222.1:46755), bd6134f31529493282c7cf50de21bfc3 (127.18.222.2:34567) I0918 20:39:22.366955 26360 tablet_bootstrap.cc:439] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3: Bootstrap starting. I0918 20:39:22.365072 26358 tablet_bootstrap.cc:439] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193: Bootstrap starting. I0918 20:39:22.389163 26359 tablet_bootstrap.cc:439] T 3cf1523b639146c29bbf465d50405209 P 184d7d5558a34378a172945529659ee6: No bootstrap required, opened a new log I0918 20:39:22.390017 26359 ts_tablet_manager.cc:1085] T 3cf1523b639146c29bbf465d50405209 P 184d7d5558a34378a172945529659ee6: Time spent bootstrapping tablet: real 0.072s user 0.008s sys 0.006s I0918 20:39:22.401405 26359 raft_consensus.cc:340] T 3cf1523b639146c29bbf465d50405209 P 184d7d5558a34378a172945529659ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } I0918 20:39:22.402549 26359 raft_consensus.cc:366] T 3cf1523b639146c29bbf465d50405209 P 184d7d5558a34378a172945529659ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:22.402885 26359 raft_consensus.cc:705] T 3cf1523b639146c29bbf465d50405209 P 184d7d5558a34378a172945529659ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184d7d5558a34378a172945529659ee6, State: Initialized, Role: FOLLOWER I0918 20:39:22.403765 26359 consensus_queue.cc:229] T 3cf1523b639146c29bbf465d50405209 P 184d7d5558a34378a172945529659ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } I0918 20:39:22.429672 26359 ts_tablet_manager.cc:1113] T 3cf1523b639146c29bbf465d50405209 P 184d7d5558a34378a172945529659ee6: Time spent starting tablet: real 0.039s user 0.006s sys 0.001s I0918 20:39:22.439345 26359 tablet_bootstrap.cc:439] T 891bd0a07cc944bfb77b70deae4de632 P 184d7d5558a34378a172945529659ee6: Bootstrap starting. I0918 20:39:22.449019 26360 tablet_bootstrap.cc:590] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3: No blocks or log segments found. Creating new log. I0918 20:39:22.444963 26358 tablet_bootstrap.cc:590] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193: No blocks or log segments found. Creating new log. I0918 20:39:22.461426 26359 tablet_bootstrap.cc:590] T 891bd0a07cc944bfb77b70deae4de632 P 184d7d5558a34378a172945529659ee6: No blocks or log segments found. Creating new log. I0918 20:39:22.494290 26358 tablet_bootstrap.cc:439] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193: No bootstrap required, opened a new log I0918 20:39:22.496758 26358 ts_tablet_manager.cc:1085] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193: Time spent bootstrapping tablet: real 0.141s user 0.015s sys 0.000s I0918 20:39:22.496568 26314 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "fe1a99b16020484a88a8f7888095479a" candidate_uuid: "184d7d5558a34378a172945529659ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4462c4818a1647ef9513032e7213c193" is_pre_election: true I0918 20:39:22.497335 26314 raft_consensus.cc:2350] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184d7d5558a34378a172945529659ee6 in term 0. I0918 20:39:22.499462 26121 leader_election.cc:310] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184d7d5558a34378a172945529659ee6, 4462c4818a1647ef9513032e7213c193; no voters: I0918 20:39:22.500421 26366 raft_consensus.cc:2644] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:22.500721 26366 raft_consensus.cc:472] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:22.501003 26366 raft_consensus.cc:2890] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:22.505959 26231 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "fe1a99b16020484a88a8f7888095479a" candidate_uuid: "4462c4818a1647ef9513032e7213c193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6134f31529493282c7cf50de21bfc3" is_pre_election: true W0918 20:39:22.507635 26278 leader_election.cc:348] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bd6134f31529493282c7cf50de21bfc3 (127.18.222.2:34567): Illegal state: must be running to vote when last-logged opid is not known I0918 20:39:22.513782 26366 raft_consensus.cc:494] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.517818 26360 tablet_bootstrap.cc:439] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3: No bootstrap required, opened a new log I0918 20:39:22.520856 26360 ts_tablet_manager.cc:1085] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3: Time spent bootstrapping tablet: real 0.154s user 0.008s sys 0.012s I0918 20:39:22.519647 26314 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "fe1a99b16020484a88a8f7888095479a" candidate_uuid: "184d7d5558a34378a172945529659ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4462c4818a1647ef9513032e7213c193" I0918 20:39:22.521405 26314 raft_consensus.cc:2890] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:22.519644 26160 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "fe1a99b16020484a88a8f7888095479a" candidate_uuid: "4462c4818a1647ef9513032e7213c193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184d7d5558a34378a172945529659ee6" is_pre_election: true I0918 20:39:22.523231 26160 raft_consensus.cc:2280] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4462c4818a1647ef9513032e7213c193 in current term 1: Already voted for candidate 184d7d5558a34378a172945529659ee6 in this term. I0918 20:39:22.518442 26366 leader_election.cc:296] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [CANDIDATE]: Term 1 election: Requested vote from peers bd6134f31529493282c7cf50de21bfc3 (127.18.222.2:34567), 4462c4818a1647ef9513032e7213c193 (127.18.222.3:39999) I0918 20:39:22.529522 26360 raft_consensus.cc:340] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.526121 26277 leader_election.cc:310] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4462c4818a1647ef9513032e7213c193; no voters: 184d7d5558a34378a172945529659ee6, bd6134f31529493282c7cf50de21bfc3 I0918 20:39:22.525506 26358 raft_consensus.cc:340] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.533793 26358 raft_consensus.cc:366] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:22.533426 26359 tablet_bootstrap.cc:439] T 891bd0a07cc944bfb77b70deae4de632 P 184d7d5558a34378a172945529659ee6: No bootstrap required, opened a new log I0918 20:39:22.533856 26314 raft_consensus.cc:2350] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184d7d5558a34378a172945529659ee6 in term 1. I0918 20:39:22.534842 26231 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "fe1a99b16020484a88a8f7888095479a" candidate_uuid: "184d7d5558a34378a172945529659ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6134f31529493282c7cf50de21bfc3" is_pre_election: true I0918 20:39:22.537286 26358 raft_consensus.cc:705] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4462c4818a1647ef9513032e7213c193, State: Initialized, Role: FOLLOWER I0918 20:39:22.535327 26230 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "fe1a99b16020484a88a8f7888095479a" candidate_uuid: "184d7d5558a34378a172945529659ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6134f31529493282c7cf50de21bfc3" I0918 20:39:22.538834 26363 raft_consensus.cc:2592] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0918 20:39:22.539125 26359 ts_tablet_manager.cc:1085] T 891bd0a07cc944bfb77b70deae4de632 P 184d7d5558a34378a172945529659ee6: Time spent bootstrapping tablet: real 0.100s user 0.007s sys 0.001s I0918 20:39:22.539505 26121 leader_election.cc:310] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184d7d5558a34378a172945529659ee6, 4462c4818a1647ef9513032e7213c193; no voters: W0918 20:39:22.540058 26124 leader_election.cc:348] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bd6134f31529493282c7cf50de21bfc3 (127.18.222.2:34567): Illegal state: must be running to vote when last-logged opid is not known W0918 20:39:22.540874 26124 leader_election.cc:348] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bd6134f31529493282c7cf50de21bfc3 (127.18.222.2:34567): Illegal state: must be running to vote when last-logged opid is not known I0918 20:39:22.542351 26358 consensus_queue.cc:229] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.547910 26366 raft_consensus.cc:2644] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:22.549863 26366 raft_consensus.cc:667] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [term 1 LEADER]: Becoming Leader. State: Replica: 184d7d5558a34378a172945529659ee6, State: Running, Role: LEADER I0918 20:39:22.550530 26366 consensus_queue.cc:206] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.552961 26360 raft_consensus.cc:366] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:22.553345 26360 raft_consensus.cc:705] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6134f31529493282c7cf50de21bfc3, State: Initialized, Role: FOLLOWER I0918 20:39:22.556814 26358 ts_tablet_manager.cc:1113] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193: Time spent starting tablet: real 0.047s user 0.007s sys 0.000s I0918 20:39:22.552455 26359 raft_consensus.cc:340] T 891bd0a07cc944bfb77b70deae4de632 P 184d7d5558a34378a172945529659ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.558706 26359 raft_consensus.cc:366] T 891bd0a07cc944bfb77b70deae4de632 P 184d7d5558a34378a172945529659ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:22.559022 26359 raft_consensus.cc:705] T 891bd0a07cc944bfb77b70deae4de632 P 184d7d5558a34378a172945529659ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184d7d5558a34378a172945529659ee6, State: Initialized, Role: FOLLOWER I0918 20:39:22.558554 26360 consensus_queue.cc:229] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.559949 26359 consensus_queue.cc:229] T 891bd0a07cc944bfb77b70deae4de632 P 184d7d5558a34378a172945529659ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.567859 26015 catalog_manager.cc:4131] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 reported cstate change: term changed from 0 to 1, leader changed from to 184d7d5558a34378a172945529659ee6 (127.18.222.1). New cstate: current_term: 1 leader_uuid: "184d7d5558a34378a172945529659ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } health_report { overall_health: UNKNOWN } } } I0918 20:39:22.573586 26363 raft_consensus.cc:472] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:22.574100 26360 ts_tablet_manager.cc:1113] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3: Time spent starting tablet: real 0.049s user 0.002s sys 0.003s I0918 20:39:22.577527 26363 raft_consensus.cc:494] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } I0918 20:39:22.583058 26363 leader_election.cc:296] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6134f31529493282c7cf50de21bfc3 (127.18.222.2:34567), 184d7d5558a34378a172945529659ee6 (127.18.222.1:46755) I0918 20:39:22.583879 26230 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3cf1523b639146c29bbf465d50405209" candidate_uuid: "4462c4818a1647ef9513032e7213c193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6134f31529493282c7cf50de21bfc3" is_pre_election: true I0918 20:39:22.584565 26160 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3cf1523b639146c29bbf465d50405209" candidate_uuid: "4462c4818a1647ef9513032e7213c193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184d7d5558a34378a172945529659ee6" is_pre_election: true I0918 20:39:22.584638 26230 raft_consensus.cc:2350] T 3cf1523b639146c29bbf465d50405209 P bd6134f31529493282c7cf50de21bfc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4462c4818a1647ef9513032e7213c193 in term 0. I0918 20:39:22.585309 26160 raft_consensus.cc:2350] T 3cf1523b639146c29bbf465d50405209 P 184d7d5558a34378a172945529659ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4462c4818a1647ef9513032e7213c193 in term 0. I0918 20:39:22.586026 26278 leader_election.cc:310] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4462c4818a1647ef9513032e7213c193, bd6134f31529493282c7cf50de21bfc3; no voters: I0918 20:39:22.586958 26363 raft_consensus.cc:2644] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:22.587409 26363 raft_consensus.cc:472] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:22.587805 26363 raft_consensus.cc:2890] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:22.588986 26359 ts_tablet_manager.cc:1113] T 891bd0a07cc944bfb77b70deae4de632 P 184d7d5558a34378a172945529659ee6: Time spent starting tablet: real 0.049s user 0.006s sys 0.001s I0918 20:39:22.592231 26360 tablet_bootstrap.cc:439] T fe1a99b16020484a88a8f7888095479a P bd6134f31529493282c7cf50de21bfc3: Bootstrap starting. I0918 20:39:22.620370 26360 tablet_bootstrap.cc:590] T fe1a99b16020484a88a8f7888095479a P bd6134f31529493282c7cf50de21bfc3: No blocks or log segments found. Creating new log. I0918 20:39:22.622225 26363 raft_consensus.cc:494] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } I0918 20:39:22.623845 26368 raft_consensus.cc:472] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:22.628288 26230 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3cf1523b639146c29bbf465d50405209" candidate_uuid: "4462c4818a1647ef9513032e7213c193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6134f31529493282c7cf50de21bfc3" I0918 20:39:22.625351 26368 raft_consensus.cc:494] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.629169 26360 tablet_bootstrap.cc:439] T fe1a99b16020484a88a8f7888095479a P bd6134f31529493282c7cf50de21bfc3: No bootstrap required, opened a new log I0918 20:39:22.629792 26360 ts_tablet_manager.cc:1085] T fe1a99b16020484a88a8f7888095479a P bd6134f31529493282c7cf50de21bfc3: Time spent bootstrapping tablet: real 0.038s user 0.010s sys 0.000s I0918 20:39:22.630486 26368 leader_election.cc:296] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184d7d5558a34378a172945529659ee6 (127.18.222.1:46755), 4462c4818a1647ef9513032e7213c193 (127.18.222.3:39999) I0918 20:39:22.629107 26230 raft_consensus.cc:2890] T 3cf1523b639146c29bbf465d50405209 P bd6134f31529493282c7cf50de21bfc3 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:22.632294 26360 raft_consensus.cc:340] T fe1a99b16020484a88a8f7888095479a P bd6134f31529493282c7cf50de21bfc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.643959 26360 raft_consensus.cc:366] T fe1a99b16020484a88a8f7888095479a P bd6134f31529493282c7cf50de21bfc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:22.644675 26360 raft_consensus.cc:705] T fe1a99b16020484a88a8f7888095479a P bd6134f31529493282c7cf50de21bfc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6134f31529493282c7cf50de21bfc3, State: Initialized, Role: FOLLOWER I0918 20:39:22.651429 26160 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3cf1523b639146c29bbf465d50405209" candidate_uuid: "4462c4818a1647ef9513032e7213c193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184d7d5558a34378a172945529659ee6" I0918 20:39:22.652287 26160 raft_consensus.cc:2890] T 3cf1523b639146c29bbf465d50405209 P 184d7d5558a34378a172945529659ee6 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:22.654496 26230 raft_consensus.cc:2350] T 3cf1523b639146c29bbf465d50405209 P bd6134f31529493282c7cf50de21bfc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4462c4818a1647ef9513032e7213c193 in term 1. I0918 20:39:22.670481 26160 raft_consensus.cc:2350] T 3cf1523b639146c29bbf465d50405209 P 184d7d5558a34378a172945529659ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4462c4818a1647ef9513032e7213c193 in term 1. I0918 20:39:22.672340 26277 leader_election.cc:310] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184d7d5558a34378a172945529659ee6, 4462c4818a1647ef9513032e7213c193; no voters: I0918 20:39:22.663589 26363 leader_election.cc:296] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 [CANDIDATE]: Term 1 election: Requested vote from peers bd6134f31529493282c7cf50de21bfc3 (127.18.222.2:34567), 184d7d5558a34378a172945529659ee6 (127.18.222.1:46755) I0918 20:39:22.679024 26360 consensus_queue.cc:229] T fe1a99b16020484a88a8f7888095479a P bd6134f31529493282c7cf50de21bfc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.683555 26363 raft_consensus.cc:2644] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:22.687377 26363 raft_consensus.cc:667] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 [term 1 LEADER]: Becoming Leader. State: Replica: 4462c4818a1647ef9513032e7213c193, State: Running, Role: LEADER I0918 20:39:22.687348 26360 ts_tablet_manager.cc:1113] T fe1a99b16020484a88a8f7888095479a P bd6134f31529493282c7cf50de21bfc3: Time spent starting tablet: real 0.057s user 0.006s sys 0.001s I0918 20:39:22.688007 26363 consensus_queue.cc:206] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } I0918 20:39:22.707289 26386 raft_consensus.cc:472] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:22.707842 26386 raft_consensus.cc:494] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.709220 26386 leader_election.cc:296] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6134f31529493282c7cf50de21bfc3 (127.18.222.2:34567), 184d7d5558a34378a172945529659ee6 (127.18.222.1:46755) I0918 20:39:22.711050 26230 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "891bd0a07cc944bfb77b70deae4de632" candidate_uuid: "4462c4818a1647ef9513032e7213c193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6134f31529493282c7cf50de21bfc3" is_pre_election: true I0918 20:39:22.711982 26230 raft_consensus.cc:2350] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4462c4818a1647ef9513032e7213c193 in term 0. I0918 20:39:22.713320 26278 leader_election.cc:310] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4462c4818a1647ef9513032e7213c193, bd6134f31529493282c7cf50de21bfc3; no voters: I0918 20:39:22.713611 26160 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "891bd0a07cc944bfb77b70deae4de632" candidate_uuid: "4462c4818a1647ef9513032e7213c193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184d7d5558a34378a172945529659ee6" is_pre_election: true I0918 20:39:22.714285 26160 raft_consensus.cc:2350] T 891bd0a07cc944bfb77b70deae4de632 P 184d7d5558a34378a172945529659ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4462c4818a1647ef9513032e7213c193 in term 0. I0918 20:39:22.722678 26314 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "891bd0a07cc944bfb77b70deae4de632" candidate_uuid: "bd6134f31529493282c7cf50de21bfc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4462c4818a1647ef9513032e7213c193" is_pre_election: true I0918 20:39:22.724287 26386 raft_consensus.cc:2644] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:22.725514 26314 raft_consensus.cc:2350] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd6134f31529493282c7cf50de21bfc3 in term 0. I0918 20:39:22.726632 26386 raft_consensus.cc:472] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:22.727035 26386 raft_consensus.cc:2890] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:22.727409 26192 leader_election.cc:310] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4462c4818a1647ef9513032e7213c193, bd6134f31529493282c7cf50de21bfc3; no voters: I0918 20:39:22.728454 26368 raft_consensus.cc:2644] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:22.728763 26368 raft_consensus.cc:472] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:22.729048 26368 raft_consensus.cc:2890] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:22.736443 26386 raft_consensus.cc:494] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.736802 26368 raft_consensus.cc:494] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.740742 26368 leader_election.cc:296] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3 [CANDIDATE]: Term 1 election: Requested vote from peers 184d7d5558a34378a172945529659ee6 (127.18.222.1:46755), 4462c4818a1647ef9513032e7213c193 (127.18.222.3:39999) I0918 20:39:22.743209 26386 leader_election.cc:296] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [CANDIDATE]: Term 1 election: Requested vote from peers bd6134f31529493282c7cf50de21bfc3 (127.18.222.2:34567), 184d7d5558a34378a172945529659ee6 (127.18.222.1:46755) I0918 20:39:22.745399 26160 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "891bd0a07cc944bfb77b70deae4de632" candidate_uuid: "4462c4818a1647ef9513032e7213c193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184d7d5558a34378a172945529659ee6" I0918 20:39:22.746794 26160 raft_consensus.cc:2890] T 891bd0a07cc944bfb77b70deae4de632 P 184d7d5558a34378a172945529659ee6 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:22.753213 26160 raft_consensus.cc:2350] T 891bd0a07cc944bfb77b70deae4de632 P 184d7d5558a34378a172945529659ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4462c4818a1647ef9513032e7213c193 in term 1. I0918 20:39:22.752871 26014 catalog_manager.cc:4131] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 reported cstate change: term changed from 0 to 1, leader changed from to 4462c4818a1647ef9513032e7213c193 (127.18.222.3). New cstate: current_term: 1 leader_uuid: "4462c4818a1647ef9513032e7213c193" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } health_report { overall_health: UNKNOWN } } } I0918 20:39:22.755668 26314 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "891bd0a07cc944bfb77b70deae4de632" candidate_uuid: "bd6134f31529493282c7cf50de21bfc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4462c4818a1647ef9513032e7213c193" I0918 20:39:22.756204 26277 leader_election.cc:310] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184d7d5558a34378a172945529659ee6, 4462c4818a1647ef9513032e7213c193; no voters: I0918 20:39:22.756566 26314 raft_consensus.cc:2280] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bd6134f31529493282c7cf50de21bfc3 in current term 1: Already voted for candidate 4462c4818a1647ef9513032e7213c193 in this term. I0918 20:39:22.761593 26230 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "891bd0a07cc944bfb77b70deae4de632" candidate_uuid: "4462c4818a1647ef9513032e7213c193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6134f31529493282c7cf50de21bfc3" I0918 20:39:22.762241 26363 raft_consensus.cc:2644] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:22.762794 26363 raft_consensus.cc:667] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [term 1 LEADER]: Becoming Leader. State: Replica: 4462c4818a1647ef9513032e7213c193, State: Running, Role: LEADER I0918 20:39:22.763445 26363 consensus_queue.cc:206] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.762315 26230 raft_consensus.cc:2280] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4462c4818a1647ef9513032e7213c193 in current term 1: Already voted for candidate bd6134f31529493282c7cf50de21bfc3 in this term. I0918 20:39:22.769093 26368 raft_consensus.cc:472] T fe1a99b16020484a88a8f7888095479a P bd6134f31529493282c7cf50de21bfc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:22.769645 26368 raft_consensus.cc:494] T fe1a99b16020484a88a8f7888095479a P bd6134f31529493282c7cf50de21bfc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } } peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } } I0918 20:39:22.772043 26368 leader_election.cc:296] T fe1a99b16020484a88a8f7888095479a P bd6134f31529493282c7cf50de21bfc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184d7d5558a34378a172945529659ee6 (127.18.222.1:46755), 4462c4818a1647ef9513032e7213c193 (127.18.222.3:39999) I0918 20:39:22.772388 26159 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "891bd0a07cc944bfb77b70deae4de632" candidate_uuid: "bd6134f31529493282c7cf50de21bfc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184d7d5558a34378a172945529659ee6" I0918 20:39:22.773358 26159 raft_consensus.cc:2280] T 891bd0a07cc944bfb77b70deae4de632 P 184d7d5558a34378a172945529659ee6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bd6134f31529493282c7cf50de21bfc3 in current term 1: Already voted for candidate 4462c4818a1647ef9513032e7213c193 in this term. I0918 20:39:22.776465 26194 leader_election.cc:310] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bd6134f31529493282c7cf50de21bfc3; no voters: 184d7d5558a34378a172945529659ee6, 4462c4818a1647ef9513032e7213c193 I0918 20:39:22.778159 26159 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "fe1a99b16020484a88a8f7888095479a" candidate_uuid: "bd6134f31529493282c7cf50de21bfc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184d7d5558a34378a172945529659ee6" is_pre_election: true I0918 20:39:22.779058 26159 raft_consensus.cc:2311] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate bd6134f31529493282c7cf50de21bfc3 for term 1 because replica is either leader or believes a valid leader to be alive. I0918 20:39:22.780361 26368 raft_consensus.cc:2592] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority I0918 20:39:22.783735 26314 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "fe1a99b16020484a88a8f7888095479a" candidate_uuid: "bd6134f31529493282c7cf50de21bfc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4462c4818a1647ef9513032e7213c193" is_pre_election: true I0918 20:39:22.784469 26314 raft_consensus.cc:2280] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bd6134f31529493282c7cf50de21bfc3 in current term 1: Already voted for candidate 184d7d5558a34378a172945529659ee6 in this term. I0918 20:39:22.789813 26192 leader_election.cc:310] T fe1a99b16020484a88a8f7888095479a P bd6134f31529493282c7cf50de21bfc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bd6134f31529493282c7cf50de21bfc3; no voters: 184d7d5558a34378a172945529659ee6, 4462c4818a1647ef9513032e7213c193 I0918 20:39:22.781457 26160 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "891bd0a07cc944bfb77b70deae4de632" candidate_uuid: "bd6134f31529493282c7cf50de21bfc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184d7d5558a34378a172945529659ee6" is_pre_election: true I0918 20:39:22.791615 26160 raft_consensus.cc:2280] T 891bd0a07cc944bfb77b70deae4de632 P 184d7d5558a34378a172945529659ee6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bd6134f31529493282c7cf50de21bfc3 in current term 1: Already voted for candidate 4462c4818a1647ef9513032e7213c193 in this term. I0918 20:39:22.802515 26014 catalog_manager.cc:4131] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 reported cstate change: term changed from 0 to 1, leader changed from to 4462c4818a1647ef9513032e7213c193 (127.18.222.3). New cstate: current_term: 1 leader_uuid: "4462c4818a1647ef9513032e7213c193" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 } health_report { overall_health: HEALTHY } } } I0918 20:39:22.807905 26368 raft_consensus.cc:2890] T fe1a99b16020484a88a8f7888095479a P bd6134f31529493282c7cf50de21bfc3 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:22.827458 26368 raft_consensus.cc:2592] T fe1a99b16020484a88a8f7888095479a P bd6134f31529493282c7cf50de21bfc3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0918 20:39:22.994976 26367 consensus_queue.cc:984] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:23.042800 26367 consensus_queue.cc:984] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4462c4818a1647ef9513032e7213c193" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 39999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:39:23.283823 26363 consensus_queue.cc:984] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:23.302835 26363 consensus_queue.cc:984] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:23.368296 26422 consensus_queue.cc:984] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184d7d5558a34378a172945529659ee6" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 46755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:23.390892 26419 consensus_queue.cc:984] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd6134f31529493282c7cf50de21bfc3" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 34567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:39:23.718710 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:23.791028 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:23.791342 19320 tablet_replica.cc:273] T 3cf1523b639146c29bbf465d50405209 P 184d7d5558a34378a172945529659ee6: stopping tablet replica I0918 20:39:23.791756 19320 raft_consensus.cc:2147] T 3cf1523b639146c29bbf465d50405209 P 184d7d5558a34378a172945529659ee6 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:23.792088 19320 raft_consensus.cc:2174] T 3cf1523b639146c29bbf465d50405209 P 184d7d5558a34378a172945529659ee6 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:23.795226 19320 tablet_replica.cc:273] T 891bd0a07cc944bfb77b70deae4de632 P 184d7d5558a34378a172945529659ee6: stopping tablet replica I0918 20:39:23.795797 19320 raft_consensus.cc:2147] T 891bd0a07cc944bfb77b70deae4de632 P 184d7d5558a34378a172945529659ee6 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:23.796087 19320 raft_consensus.cc:2174] T 891bd0a07cc944bfb77b70deae4de632 P 184d7d5558a34378a172945529659ee6 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:23.800036 19320 tablet_replica.cc:273] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6: stopping tablet replica I0918 20:39:23.800463 19320 raft_consensus.cc:2147] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:23.802171 19320 raft_consensus.cc:2174] T fe1a99b16020484a88a8f7888095479a P 184d7d5558a34378a172945529659ee6 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:23.836504 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:23.836747 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:23.837188 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:23.850934 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:23.874727 26192 messenger.cc:381] Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer W0918 20:39:23.877382 26278 consensus_peers.cc:458] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 -> Peer bd6134f31529493282c7cf50de21bfc3 (127.18.222.2:34567): Couldn't send request to peer bd6134f31529493282c7cf50de21bfc3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry. I0918 20:39:23.903898 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:23.904271 19320 tablet_replica.cc:273] T fe1a99b16020484a88a8f7888095479a P bd6134f31529493282c7cf50de21bfc3: stopping tablet replica I0918 20:39:23.904695 19320 raft_consensus.cc:2147] T fe1a99b16020484a88a8f7888095479a P bd6134f31529493282c7cf50de21bfc3 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:23.905009 19320 raft_consensus.cc:2174] T fe1a99b16020484a88a8f7888095479a P bd6134f31529493282c7cf50de21bfc3 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:23.907953 19320 tablet_replica.cc:273] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3: stopping tablet replica I0918 20:39:23.908504 19320 raft_consensus.cc:2147] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:23.908790 19320 raft_consensus.cc:2174] T 891bd0a07cc944bfb77b70deae4de632 P bd6134f31529493282c7cf50de21bfc3 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:23.912154 19320 tablet_replica.cc:273] T 3cf1523b639146c29bbf465d50405209 P bd6134f31529493282c7cf50de21bfc3: stopping tablet replica I0918 20:39:23.912539 19320 raft_consensus.cc:2147] T 3cf1523b639146c29bbf465d50405209 P bd6134f31529493282c7cf50de21bfc3 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:23.912816 19320 raft_consensus.cc:2174] T 3cf1523b639146c29bbf465d50405209 P bd6134f31529493282c7cf50de21bfc3 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:23.928203 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:23.928406 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:23.928746 19320 tablet_server.cc:169] TabletServer shutdown complete. W0918 20:39:23.934090 26277 consensus_peers.cc:458] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 -> Peer 184d7d5558a34378a172945529659ee6 (127.18.222.1:46755): Couldn't send request to peer 184d7d5558a34378a172945529659ee6. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.1:46755: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0918 20:39:23.934459 26277 consensus_peers.cc:458] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 -> Peer 184d7d5558a34378a172945529659ee6 (127.18.222.1:46755): Couldn't send request to peer 184d7d5558a34378a172945529659ee6. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.1:46755: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:39:23.942755 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:23.972781 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:23.973170 19320 tablet_replica.cc:273] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193: stopping tablet replica I0918 20:39:23.973678 19320 raft_consensus.cc:2147] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:23.974584 19320 raft_consensus.cc:2174] T 3cf1523b639146c29bbf465d50405209 P 4462c4818a1647ef9513032e7213c193 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:23.977552 19320 tablet_replica.cc:273] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193: stopping tablet replica I0918 20:39:23.977936 19320 raft_consensus.cc:2147] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:23.978555 19320 raft_consensus.cc:2174] T 891bd0a07cc944bfb77b70deae4de632 P 4462c4818a1647ef9513032e7213c193 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:23.980970 19320 tablet_replica.cc:273] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193: stopping tablet replica I0918 20:39:23.981338 19320 raft_consensus.cc:2147] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:23.981629 19320 raft_consensus.cc:2174] T fe1a99b16020484a88a8f7888095479a P 4462c4818a1647ef9513032e7213c193 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:23.999548 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:23.999804 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:24.000161 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:24.011034 19320 master.cc:247] Master@127.18.222.62:34959 shutting down... I0918 20:39:24.031155 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:24.031534 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:24.031859 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P b27de89bfdca4ef4a3baf93a01c8d303: stopping tablet replica I0918 20:39:24.050093 19320 master.cc:258] Master@127.18.222.62:34959 shutdown complete. I0918 20:39:24.059960 19320 master.cc:247] Master@127.18.222.61:43721 shutting down... I0918 20:39:24.077821 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:24.078594 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:24.078910 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P b1328916d51a41baa01800e39f6e6ad2: stopping tablet replica I0918 20:39:24.097332 19320 master.cc:258] Master@127.18.222.61:43721 shutdown complete. I0918 20:39:24.107810 19320 master.cc:247] Master@127.18.222.60:33177 shutting down... I0918 20:39:24.129796 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:24.130157 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:24.130367 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 3026503f76814232b3a196172349ca5c: stopping tablet replica I0918 20:39:24.148452 19320 master.cc:258] Master@127.18.222.60:33177 shutdown complete. [ OK ] RemoteKsckTest.TestNonMatchingTableFilter (3427 ms) [ RUN ] RemoteKsckTest.TestMatchingTableFilter I0918 20:39:24.181779 19320 test_util.cc:212] Using random seed: -719897355 I0918 20:39:24.182611 19320 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.18.222.62:38661,127.18.222.61:43053,127.18.222.60:45059 I0918 20:39:24.195854 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 159592us I0918 20:39:24.196244 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:24.196409 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:24.196660 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:24.196784 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:24.197878 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/master-0-root/instance: uuid: "8093d7d8c08b46af9003e7ae02eea403" format_stamp: "Formatted at 2019-09-18 20:39:24 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:24.206566 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.010s sys 0.001s I0918 20:39:24.208118 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:24.214684 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:24.214860 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/master-0-root uuid: "8093d7d8c08b46af9003e7ae02eea403" format_stamp: "Formatted at 2019-09-18 20:39:24 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:24.215147 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:24.264178 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:38661 I0918 20:39:24.265974 19320 webserver.cc:374] Webserver started at http://127.18.222.62:36669/ using document root and password file I0918 20:39:24.267797 26495 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:24.272899 26495 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:24.286098 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 159592us I0918 20:39:24.286581 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:24.286759 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:24.286978 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:24.287109 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:24.287915 26495 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:24.289840 26495 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:43053: Network error: Client connection negotiation failed: client connection to 127.18.222.61:43053: connect: Connection refused (error 111) I0918 20:39:24.290087 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/master-1-root/instance: uuid: "b365daeaabca4390bae27ba6253826cf" format_stamp: "Formatted at 2019-09-18 20:39:24 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:24.299938 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s I0918 20:39:24.301440 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:24.309150 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:24.309394 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/master-1-root uuid: "b365daeaabca4390bae27ba6253826cf" format_stamp: "Formatted at 2019-09-18 20:39:24 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:24.309655 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:24.327237 26495 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } attempt: 1 W0918 20:39:24.329437 26495 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:43053: Network error: Client connection negotiation failed: client connection to 127.18.222.61:43053: connect: Connection refused (error 111) I0918 20:39:24.377374 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:43053 I0918 20:39:24.379369 19320 webserver.cc:374] Webserver started at http://127.18.222.61:37243/ using document root and password file I0918 20:39:24.381603 26552 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:24.388566 26552 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:24.393813 26495 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } attempt: 2 I0918 20:39:24.401369 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 159592us I0918 20:39:24.401968 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:24.402209 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:24.402500 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:24.402715 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:24.403676 26552 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:24.404072 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/master-2-root/instance: uuid: "2a4885def8064583a05ff501f6773dc8" format_stamp: "Formatted at 2019-09-18 20:39:24 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:24.412129 26495 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:24.414237 26495 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:45059: Network error: Client connection negotiation failed: client connection to 127.18.222.60:45059: connect: Connection refused (error 111) I0918 20:39:24.417510 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.013s user 0.009s sys 0.004s I0918 20:39:24.419636 26552 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:24.419775 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 W0918 20:39:24.421483 26552 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:45059: Network error: Client connection negotiation failed: client connection to 127.18.222.60:45059: connect: Connection refused (error 111) I0918 20:39:24.428400 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:24.428686 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/master-2-root uuid: "2a4885def8064583a05ff501f6773dc8" format_stamp: "Formatted at 2019-09-18 20:39:24 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:24.428970 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:24.432710 26495 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } attempt: 1 W0918 20:39:24.435029 26495 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:45059: Network error: Client connection negotiation failed: client connection to 127.18.222.60:45059: connect: Connection refused (error 111) I0918 20:39:24.441929 26552 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } attempt: 1 W0918 20:39:24.444059 26552 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:45059: Network error: Client connection negotiation failed: client connection to 127.18.222.60:45059: connect: Connection refused (error 111) I0918 20:39:24.479739 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:45059 I0918 20:39:24.480500 26552 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } attempt: 2 I0918 20:39:24.481832 19320 webserver.cc:374] Webserver started at http://127.18.222.60:36533/ using document root and password file I0918 20:39:24.484102 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0918 20:39:24.485355 26611 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:24.489475 26495 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } attempt: 2 I0918 20:39:24.493067 26611 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:24.503018 26552 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf: Bootstrap starting. I0918 20:39:24.507838 26552 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf: No blocks or log segments found. Creating new log. I0918 20:39:24.511487 26611 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:24.514588 26495 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403: Bootstrap starting. I0918 20:39:24.518290 26495 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403: No blocks or log segments found. Creating new log. I0918 20:39:24.521780 26552 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf: No bootstrap required, opened a new log I0918 20:39:24.523247 26495 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403: No bootstrap required, opened a new log I0918 20:39:24.524435 26552 raft_consensus.cc:340] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } I0918 20:39:24.525326 26552 raft_consensus.cc:366] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:24.525617 26552 raft_consensus.cc:705] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b365daeaabca4390bae27ba6253826cf, State: Initialized, Role: FOLLOWER I0918 20:39:24.525530 26495 raft_consensus.cc:340] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } I0918 20:39:24.526537 26495 raft_consensus.cc:366] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:24.526830 26495 raft_consensus.cc:705] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8093d7d8c08b46af9003e7ae02eea403, State: Initialized, Role: FOLLOWER I0918 20:39:24.526715 26552 consensus_queue.cc:229] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } I0918 20:39:24.527451 26611 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:24.527767 26495 consensus_queue.cc:229] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } I0918 20:39:24.530189 26552 sys_catalog.cc:340] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } } I0918 20:39:24.531258 26552 sys_catalog.cc:343] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:24.530642 26616 sys_catalog.cc:340] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } } I0918 20:39:24.532335 26616 sys_catalog.cc:343] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:24.532042 26495 sys_catalog.cc:340] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } } I0918 20:39:24.532904 26495 sys_catalog.cc:343] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:24.532954 26552 sys_catalog.cc:428] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:24.533105 26617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } } I0918 20:39:24.534087 26617 sys_catalog.cc:343] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:24.534250 26495 sys_catalog.cc:428] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:24.537418 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 I0918 20:39:24.537683 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 W0918 20:39:24.540721 26618 catalog_manager.cc:1155] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:24.541334 26618 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0918 20:39:24.540721 26619 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:24.541676 26619 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:24.546777 26611 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8: Bootstrap starting. I0918 20:39:24.549948 26611 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8: No blocks or log segments found. Creating new log. I0918 20:39:24.550061 26616 raft_consensus.cc:472] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:24.550583 26616 raft_consensus.cc:494] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } I0918 20:39:24.552798 26616 leader_election.cc:296] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8093d7d8c08b46af9003e7ae02eea403 (127.18.222.62:38661), 2a4885def8064583a05ff501f6773dc8 (127.18.222.60:45059) I0918 20:39:24.553627 26472 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b365daeaabca4390bae27ba6253826cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8093d7d8c08b46af9003e7ae02eea403" is_pre_election: true I0918 20:39:24.554457 26611 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8: No bootstrap required, opened a new log I0918 20:39:24.554433 26472 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b365daeaabca4390bae27ba6253826cf in term 0. I0918 20:39:24.555550 26503 leader_election.cc:310] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8093d7d8c08b46af9003e7ae02eea403, b365daeaabca4390bae27ba6253826cf; no voters: I0918 20:39:24.556537 26616 raft_consensus.cc:2644] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:24.556834 26616 raft_consensus.cc:472] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:24.557152 26616 raft_consensus.cc:2890] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:24.556937 26611 raft_consensus.cc:340] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } I0918 20:39:24.557615 26611 raft_consensus.cc:366] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:24.557871 26611 raft_consensus.cc:705] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a4885def8064583a05ff501f6773dc8, State: Initialized, Role: FOLLOWER I0918 20:39:24.558513 26611 consensus_queue.cc:229] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } I0918 20:39:24.561111 26611 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } } I0918 20:39:24.561851 26611 sys_catalog.cc:343] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:24.561923 26616 raft_consensus.cc:494] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } I0918 20:39:24.563421 26611 sys_catalog.cc:428] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:24.561733 26621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } } I0918 20:39:24.563515 26587 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b365daeaabca4390bae27ba6253826cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a4885def8064583a05ff501f6773dc8" is_pre_election: true I0918 20:39:24.563913 26616 leader_election.cc:296] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [CANDIDATE]: Term 1 election: Requested vote from peers 8093d7d8c08b46af9003e7ae02eea403 (127.18.222.62:38661), 2a4885def8064583a05ff501f6773dc8 (127.18.222.60:45059) I0918 20:39:24.564141 26621 sys_catalog.cc:343] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:24.564821 26587 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b365daeaabca4390bae27ba6253826cf in term 0. I0918 20:39:24.564509 26472 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b365daeaabca4390bae27ba6253826cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8093d7d8c08b46af9003e7ae02eea403" I0918 20:39:24.564864 26586 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b365daeaabca4390bae27ba6253826cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a4885def8064583a05ff501f6773dc8" I0918 20:39:24.565294 26472 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:24.565490 26586 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:24.571568 26472 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b365daeaabca4390bae27ba6253826cf in term 1. I0918 20:39:24.571568 26586 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b365daeaabca4390bae27ba6253826cf in term 1. I0918 20:39:24.572899 26503 leader_election.cc:310] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8093d7d8c08b46af9003e7ae02eea403, b365daeaabca4390bae27ba6253826cf; no voters: I0918 20:39:24.574211 26616 raft_consensus.cc:2644] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [term 1 FOLLOWER]: Leader election won for term 1 W0918 20:39:24.576287 26622 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:24.576696 26622 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:24.576984 26616 raft_consensus.cc:667] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [term 1 LEADER]: Becoming Leader. State: Replica: b365daeaabca4390bae27ba6253826cf, State: Running, Role: LEADER I0918 20:39:24.577544 26616 consensus_queue.cc:206] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } I0918 20:39:24.580880 26623 sys_catalog.cc:340] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b365daeaabca4390bae27ba6253826cf. Latest consensus state: current_term: 1 leader_uuid: "b365daeaabca4390bae27ba6253826cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } } I0918 20:39:24.581621 26623 sys_catalog.cc:343] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [sys.catalog]: This master's current role is: LEADER I0918 20:39:24.583144 26625 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:39:24.586861 26625 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:39:24.587031 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 159592us I0918 20:39:24.587572 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:24.587823 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:24.588115 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:24.588284 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:24.589622 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-0-root/instance: uuid: "5b818b6c74f54250b78db6a86557cddb" format_stamp: "Formatted at 2019-09-18 20:39:24 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:24.609136 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.019s user 0.011s sys 0.009s I0918 20:39:24.611515 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:24.625792 19320 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s I0918 20:39:24.626027 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-0-root uuid: "5b818b6c74f54250b78db6a86557cddb" format_stamp: "Formatted at 2019-09-18 20:39:24 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:24.626291 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:24.630086 26472 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403 [term 1 FOLLOWER]: Refusing update from remote peer b365daeaabca4390bae27ba6253826cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:24.630012 26586 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8 [term 1 FOLLOWER]: Refusing update from remote peer b365daeaabca4390bae27ba6253826cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:24.632558 26616 consensus_queue.cc:984] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:24.633319 26623 consensus_queue.cc:984] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:24.643537 26621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b365daeaabca4390bae27ba6253826cf. Latest consensus state: current_term: 1 leader_uuid: "b365daeaabca4390bae27ba6253826cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } } I0918 20:39:24.644240 26621 sys_catalog.cc:343] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:24.648382 26623 sys_catalog.cc:340] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b365daeaabca4390bae27ba6253826cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } } I0918 20:39:24.649583 26623 sys_catalog.cc:343] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [sys.catalog]: This master's current role is: LEADER I0918 20:39:24.650508 26625 catalog_manager.cc:981] Generated new certificate authority record I0918 20:39:24.651085 26617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b365daeaabca4390bae27ba6253826cf. Latest consensus state: current_term: 1 leader_uuid: "b365daeaabca4390bae27ba6253826cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } } I0918 20:39:24.651895 26617 sys_catalog.cc:343] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:24.654273 26625 catalog_manager.cc:1106] Loading token signing keys... I0918 20:39:24.666596 26621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b365daeaabca4390bae27ba6253826cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } } I0918 20:39:24.667675 26621 sys_catalog.cc:343] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:24.661900 26623 sys_catalog.cc:340] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b365daeaabca4390bae27ba6253826cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } } I0918 20:39:24.668293 26623 sys_catalog.cc:343] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [sys.catalog]: This master's current role is: LEADER I0918 20:39:24.675861 26617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b365daeaabca4390bae27ba6253826cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8093d7d8c08b46af9003e7ae02eea403" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 38661 } } peers { permanent_uuid: "b365daeaabca4390bae27ba6253826cf" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 43053 } } peers { permanent_uuid: "2a4885def8064583a05ff501f6773dc8" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 45059 } } } I0918 20:39:24.676517 26617 sys_catalog.cc:343] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:24.681428 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:24.681697 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:24.694924 26625 catalog_manager.cc:4446] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf: Generated new TSK 0 I0918 20:39:24.695971 26625 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:39:24.734251 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:41007 I0918 20:39:24.736838 19320 webserver.cc:374] Webserver started at http://127.18.222.1:43961/ using document root and password file I0918 20:39:24.758910 26699 heartbeater.cc:322] Connected to a master server at 127.18.222.60:45059 I0918 20:39:24.759326 26699 heartbeater.cc:406] Registering TS with master... I0918 20:39:24.760016 26699 heartbeater.cc:455] Master 127.18.222.60:45059 requested a full tablet report, sending... I0918 20:39:24.763168 26577 ts_manager.cc:186] Registered new tserver with Master: 5b818b6c74f54250b78db6a86557cddb (127.18.222.1:41007) I0918 20:39:24.766852 26701 heartbeater.cc:322] Connected to a master server at 127.18.222.62:38661 I0918 20:39:24.767210 26701 heartbeater.cc:406] Registering TS with master... I0918 20:39:24.767431 26700 heartbeater.cc:322] Connected to a master server at 127.18.222.61:43053 I0918 20:39:24.767889 26700 heartbeater.cc:406] Registering TS with master... I0918 20:39:24.767971 26701 heartbeater.cc:455] Master 127.18.222.62:38661 requested a full tablet report, sending... I0918 20:39:24.768404 26700 heartbeater.cc:455] Master 127.18.222.61:43053 requested a full tablet report, sending... I0918 20:39:24.770267 26519 ts_manager.cc:186] Registered new tserver with Master: 5b818b6c74f54250b78db6a86557cddb (127.18.222.1:41007) I0918 20:39:24.770834 26462 ts_manager.cc:186] Registered new tserver with Master: 5b818b6c74f54250b78db6a86557cddb (127.18.222.1:41007) I0918 20:39:24.772897 26519 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42186 I0918 20:39:24.774127 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 159592us I0918 20:39:24.774509 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:24.774682 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:24.774911 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:24.775048 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:24.776262 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-1-root/instance: uuid: "9ed908683a4b4657aed8cbac7c7db7fd" format_stamp: "Formatted at 2019-09-18 20:39:24 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:24.785892 26700 heartbeater.cc:447] Master 127.18.222.61:43053 was elected leader, sending a full tablet report... I0918 20:39:24.787258 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.002s sys 0.009s I0918 20:39:24.789347 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:24.798504 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0918 20:39:24.798756 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-1-root uuid: "9ed908683a4b4657aed8cbac7c7db7fd" format_stamp: "Formatted at 2019-09-18 20:39:24 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:24.799010 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:24.853705 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:24.853999 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:24.901304 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.2:38505 I0918 20:39:24.905254 19320 webserver.cc:374] Webserver started at http://127.18.222.2:43669/ using document root and password file I0918 20:39:24.943389 26770 heartbeater.cc:322] Connected to a master server at 127.18.222.60:45059 I0918 20:39:24.943830 26770 heartbeater.cc:406] Registering TS with master... I0918 20:39:24.944438 26770 heartbeater.cc:455] Master 127.18.222.60:45059 requested a full tablet report, sending... I0918 20:39:24.947204 26577 ts_manager.cc:186] Registered new tserver with Master: 9ed908683a4b4657aed8cbac7c7db7fd (127.18.222.2:38505) I0918 20:39:24.955320 26771 heartbeater.cc:322] Connected to a master server at 127.18.222.61:43053 I0918 20:39:24.955749 26771 heartbeater.cc:406] Registering TS with master... I0918 20:39:24.956045 26772 heartbeater.cc:322] Connected to a master server at 127.18.222.62:38661 I0918 20:39:24.956388 26772 heartbeater.cc:406] Registering TS with master... I0918 20:39:24.956449 26771 heartbeater.cc:455] Master 127.18.222.61:43053 requested a full tablet report, sending... I0918 20:39:24.956792 26772 heartbeater.cc:455] Master 127.18.222.62:38661 requested a full tablet report, sending... I0918 20:39:24.958389 26519 ts_manager.cc:186] Registered new tserver with Master: 9ed908683a4b4657aed8cbac7c7db7fd (127.18.222.2:38505) I0918 20:39:24.959172 26462 ts_manager.cc:186] Registered new tserver with Master: 9ed908683a4b4657aed8cbac7c7db7fd (127.18.222.2:38505) I0918 20:39:24.960021 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 159592us I0918 20:39:24.960333 26519 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42192 I0918 20:39:24.960532 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:24.960855 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:24.961197 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:24.961386 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:24.962946 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-2-root/instance: uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" format_stamp: "Formatted at 2019-09-18 20:39:24 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:24.976085 26771 heartbeater.cc:447] Master 127.18.222.61:43053 was elected leader, sending a full tablet report... I0918 20:39:24.978134 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.015s user 0.008s sys 0.005s I0918 20:39:24.980924 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:24.990375 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:24.990677 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-2-root uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" format_stamp: "Formatted at 2019-09-18 20:39:24 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:24.991032 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1568839089958813-19320/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:25.030328 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:25.030573 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:25.097224 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.3:45467 I0918 20:39:25.105289 19320 webserver.cc:374] Webserver started at http://127.18.222.3:44961/ using document root and password file I0918 20:39:25.153937 26841 heartbeater.cc:322] Connected to a master server at 127.18.222.60:45059 I0918 20:39:25.154328 26841 heartbeater.cc:406] Registering TS with master... I0918 20:39:25.154925 26841 heartbeater.cc:455] Master 127.18.222.60:45059 requested a full tablet report, sending... I0918 20:39:25.157871 26844 heartbeater.cc:322] Connected to a master server at 127.18.222.62:38661 I0918 20:39:25.158211 26844 heartbeater.cc:406] Registering TS with master... I0918 20:39:25.158054 26576 ts_manager.cc:186] Registered new tserver with Master: 2e5fe8bb38b645979eb1c8da1c6f5e98 (127.18.222.3:45467) I0918 20:39:25.161356 26844 heartbeater.cc:455] Master 127.18.222.62:38661 requested a full tablet report, sending... I0918 20:39:25.163035 26842 heartbeater.cc:322] Connected to a master server at 127.18.222.61:43053 I0918 20:39:25.163345 26842 heartbeater.cc:406] Registering TS with master... I0918 20:39:25.163841 26842 heartbeater.cc:455] Master 127.18.222.61:43053 requested a full tablet report, sending... I0918 20:39:25.163784 26461 ts_manager.cc:186] Registered new tserver with Master: 2e5fe8bb38b645979eb1c8da1c6f5e98 (127.18.222.3:45467) I0918 20:39:25.165627 26519 ts_manager.cc:186] Registered new tserver with Master: 2e5fe8bb38b645979eb1c8da1c6f5e98 (127.18.222.3:45467) I0918 20:39:25.165946 19320 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.02864798s I0918 20:39:25.167376 26519 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42198 I0918 20:39:25.181234 26842 heartbeater.cc:447] Master 127.18.222.61:43053 was elected leader, sending a full tablet report... I0918 20:39:25.286273 26518 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42206: name: "ksck-test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\n\000\000\000\004\001\024\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:39:25.287991 26518 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:39:25.457146 26666 tablet_service.cc:1022] Processing CreateTablet for tablet 8d5464716606431684b2b9eec5ec1a24 (table=ksck-test-table [id=c03bb60bcc854aac803eb91272c843c4]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:25.458568 26666 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 8d5464716606431684b2b9eec5ec1a24. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:25.461253 26665 tablet_service.cc:1022] Processing CreateTablet for tablet 6d805ab6e7654ea99bd1e9e6e57217cc (table=ksck-test-table [id=c03bb60bcc854aac803eb91272c843c4]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:25.462555 26665 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 6d805ab6e7654ea99bd1e9e6e57217cc. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:25.472960 26664 tablet_service.cc:1022] Processing CreateTablet for tablet 61c6e28dc1684699a5ed860fc8a6201c (table=ksck-test-table [id=c03bb60bcc854aac803eb91272c843c4]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:25.474200 26664 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 61c6e28dc1684699a5ed860fc8a6201c. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:25.496496 26737 tablet_service.cc:1022] Processing CreateTablet for tablet 8d5464716606431684b2b9eec5ec1a24 (table=ksck-test-table [id=c03bb60bcc854aac803eb91272c843c4]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:25.497757 26737 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 8d5464716606431684b2b9eec5ec1a24. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:25.499945 26736 tablet_service.cc:1022] Processing CreateTablet for tablet 6d805ab6e7654ea99bd1e9e6e57217cc (table=ksck-test-table [id=c03bb60bcc854aac803eb91272c843c4]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:25.501019 26736 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 6d805ab6e7654ea99bd1e9e6e57217cc. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:25.505416 26735 tablet_service.cc:1022] Processing CreateTablet for tablet 61c6e28dc1684699a5ed860fc8a6201c (table=ksck-test-table [id=c03bb60bcc854aac803eb91272c843c4]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:25.506583 26735 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 61c6e28dc1684699a5ed860fc8a6201c. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:25.509311 26808 tablet_service.cc:1022] Processing CreateTablet for tablet 8d5464716606431684b2b9eec5ec1a24 (table=ksck-test-table [id=c03bb60bcc854aac803eb91272c843c4]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:25.510589 26808 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 8d5464716606431684b2b9eec5ec1a24. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:25.523419 26807 tablet_service.cc:1022] Processing CreateTablet for tablet 6d805ab6e7654ea99bd1e9e6e57217cc (table=ksck-test-table [id=c03bb60bcc854aac803eb91272c843c4]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:25.524896 26807 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 6d805ab6e7654ea99bd1e9e6e57217cc. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:25.537647 26806 tablet_service.cc:1022] Processing CreateTablet for tablet 61c6e28dc1684699a5ed860fc8a6201c (table=ksck-test-table [id=c03bb60bcc854aac803eb91272c843c4]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:25.538846 26806 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 61c6e28dc1684699a5ed860fc8a6201c. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:25.561110 26619 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403: acquiring CA information for follower catalog manager: success I0918 20:39:25.566920 26619 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:25.579293 26870 tablet_bootstrap.cc:439] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb: Bootstrap starting. I0918 20:39:25.586846 26870 tablet_bootstrap.cc:590] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb: No blocks or log segments found. Creating new log. I0918 20:39:25.587216 26622 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8: acquiring CA information for follower catalog manager: success I0918 20:39:25.589881 26622 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:25.600208 26871 tablet_bootstrap.cc:439] T 6d805ab6e7654ea99bd1e9e6e57217cc P 9ed908683a4b4657aed8cbac7c7db7fd: Bootstrap starting. I0918 20:39:25.603127 26871 tablet_bootstrap.cc:590] T 6d805ab6e7654ea99bd1e9e6e57217cc P 9ed908683a4b4657aed8cbac7c7db7fd: No blocks or log segments found. Creating new log. I0918 20:39:25.603935 26872 tablet_bootstrap.cc:439] T 8d5464716606431684b2b9eec5ec1a24 P 2e5fe8bb38b645979eb1c8da1c6f5e98: Bootstrap starting. I0918 20:39:25.607149 26872 tablet_bootstrap.cc:590] T 8d5464716606431684b2b9eec5ec1a24 P 2e5fe8bb38b645979eb1c8da1c6f5e98: No blocks or log segments found. Creating new log. I0918 20:39:25.616479 26870 tablet_bootstrap.cc:439] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb: No bootstrap required, opened a new log I0918 20:39:25.617101 26870 ts_tablet_manager.cc:1085] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.000s I0918 20:39:25.619582 26870 raft_consensus.cc:340] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.620467 26870 raft_consensus.cc:366] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:25.620730 26870 raft_consensus.cc:705] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b818b6c74f54250b78db6a86557cddb, State: Initialized, Role: FOLLOWER I0918 20:39:25.621596 26870 consensus_queue.cc:229] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.635088 26872 tablet_bootstrap.cc:439] T 8d5464716606431684b2b9eec5ec1a24 P 2e5fe8bb38b645979eb1c8da1c6f5e98: No bootstrap required, opened a new log I0918 20:39:25.635850 26872 ts_tablet_manager.cc:1085] T 8d5464716606431684b2b9eec5ec1a24 P 2e5fe8bb38b645979eb1c8da1c6f5e98: Time spent bootstrapping tablet: real 0.032s user 0.015s sys 0.004s I0918 20:39:25.635382 26871 tablet_bootstrap.cc:439] T 6d805ab6e7654ea99bd1e9e6e57217cc P 9ed908683a4b4657aed8cbac7c7db7fd: No bootstrap required, opened a new log I0918 20:39:25.643471 26871 ts_tablet_manager.cc:1085] T 6d805ab6e7654ea99bd1e9e6e57217cc P 9ed908683a4b4657aed8cbac7c7db7fd: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.000s I0918 20:39:25.648156 26872 raft_consensus.cc:340] T 8d5464716606431684b2b9eec5ec1a24 P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.649160 26872 raft_consensus.cc:366] T 8d5464716606431684b2b9eec5ec1a24 P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:25.649579 26872 raft_consensus.cc:705] T 8d5464716606431684b2b9eec5ec1a24 P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5fe8bb38b645979eb1c8da1c6f5e98, State: Initialized, Role: FOLLOWER I0918 20:39:25.649142 26871 raft_consensus.cc:340] T 6d805ab6e7654ea99bd1e9e6e57217cc P 9ed908683a4b4657aed8cbac7c7db7fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.650194 26871 raft_consensus.cc:366] T 6d805ab6e7654ea99bd1e9e6e57217cc P 9ed908683a4b4657aed8cbac7c7db7fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:25.650502 26871 raft_consensus.cc:705] T 6d805ab6e7654ea99bd1e9e6e57217cc P 9ed908683a4b4657aed8cbac7c7db7fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed908683a4b4657aed8cbac7c7db7fd, State: Initialized, Role: FOLLOWER I0918 20:39:25.650375 26872 consensus_queue.cc:229] T 8d5464716606431684b2b9eec5ec1a24 P 2e5fe8bb38b645979eb1c8da1c6f5e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.651824 26870 ts_tablet_manager.cc:1113] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb: Time spent starting tablet: real 0.034s user 0.007s sys 0.008s I0918 20:39:25.651417 26871 consensus_queue.cc:229] T 6d805ab6e7654ea99bd1e9e6e57217cc P 9ed908683a4b4657aed8cbac7c7db7fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.652537 26870 tablet_bootstrap.cc:439] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb: Bootstrap starting. I0918 20:39:25.656719 26870 tablet_bootstrap.cc:590] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb: No blocks or log segments found. Creating new log. I0918 20:39:25.677434 26871 ts_tablet_manager.cc:1113] T 6d805ab6e7654ea99bd1e9e6e57217cc P 9ed908683a4b4657aed8cbac7c7db7fd: Time spent starting tablet: real 0.034s user 0.011s sys 0.000s I0918 20:39:25.678365 26871 tablet_bootstrap.cc:439] T 61c6e28dc1684699a5ed860fc8a6201c P 9ed908683a4b4657aed8cbac7c7db7fd: Bootstrap starting. I0918 20:39:25.681967 26871 tablet_bootstrap.cc:590] T 61c6e28dc1684699a5ed860fc8a6201c P 9ed908683a4b4657aed8cbac7c7db7fd: No blocks or log segments found. Creating new log. I0918 20:39:25.707407 26870 tablet_bootstrap.cc:439] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb: No bootstrap required, opened a new log I0918 20:39:25.708050 26870 ts_tablet_manager.cc:1085] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb: Time spent bootstrapping tablet: real 0.056s user 0.014s sys 0.005s I0918 20:39:25.710343 26870 raft_consensus.cc:340] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.711061 26870 raft_consensus.cc:366] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:25.711306 26870 raft_consensus.cc:705] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b818b6c74f54250b78db6a86557cddb, State: Initialized, Role: FOLLOWER I0918 20:39:25.711942 26870 consensus_queue.cc:229] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.713404 26870 ts_tablet_manager.cc:1113] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s I0918 20:39:25.714562 26870 tablet_bootstrap.cc:439] T 8d5464716606431684b2b9eec5ec1a24 P 5b818b6c74f54250b78db6a86557cddb: Bootstrap starting. I0918 20:39:25.720177 26871 tablet_bootstrap.cc:439] T 61c6e28dc1684699a5ed860fc8a6201c P 9ed908683a4b4657aed8cbac7c7db7fd: No bootstrap required, opened a new log I0918 20:39:25.720911 26871 ts_tablet_manager.cc:1085] T 61c6e28dc1684699a5ed860fc8a6201c P 9ed908683a4b4657aed8cbac7c7db7fd: Time spent bootstrapping tablet: real 0.043s user 0.017s sys 0.007s I0918 20:39:25.721525 26870 tablet_bootstrap.cc:590] T 8d5464716606431684b2b9eec5ec1a24 P 5b818b6c74f54250b78db6a86557cddb: No blocks or log segments found. Creating new log. I0918 20:39:25.724907 26871 raft_consensus.cc:340] T 61c6e28dc1684699a5ed860fc8a6201c P 9ed908683a4b4657aed8cbac7c7db7fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.725860 26871 raft_consensus.cc:366] T 61c6e28dc1684699a5ed860fc8a6201c P 9ed908683a4b4657aed8cbac7c7db7fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:25.726059 26871 raft_consensus.cc:705] T 61c6e28dc1684699a5ed860fc8a6201c P 9ed908683a4b4657aed8cbac7c7db7fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed908683a4b4657aed8cbac7c7db7fd, State: Initialized, Role: FOLLOWER I0918 20:39:25.726899 26871 consensus_queue.cc:229] T 61c6e28dc1684699a5ed860fc8a6201c P 9ed908683a4b4657aed8cbac7c7db7fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.729401 26871 ts_tablet_manager.cc:1113] T 61c6e28dc1684699a5ed860fc8a6201c P 9ed908683a4b4657aed8cbac7c7db7fd: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s I0918 20:39:25.735446 26871 tablet_bootstrap.cc:439] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd: Bootstrap starting. I0918 20:39:25.736886 26872 ts_tablet_manager.cc:1113] T 8d5464716606431684b2b9eec5ec1a24 P 2e5fe8bb38b645979eb1c8da1c6f5e98: Time spent starting tablet: real 0.101s user 0.023s sys 0.014s I0918 20:39:25.737857 26872 tablet_bootstrap.cc:439] T 61c6e28dc1684699a5ed860fc8a6201c P 2e5fe8bb38b645979eb1c8da1c6f5e98: Bootstrap starting. I0918 20:39:25.746151 26870 tablet_bootstrap.cc:439] T 8d5464716606431684b2b9eec5ec1a24 P 5b818b6c74f54250b78db6a86557cddb: No bootstrap required, opened a new log I0918 20:39:25.746788 26870 ts_tablet_manager.cc:1085] T 8d5464716606431684b2b9eec5ec1a24 P 5b818b6c74f54250b78db6a86557cddb: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.000s I0918 20:39:25.748800 26870 raft_consensus.cc:340] T 8d5464716606431684b2b9eec5ec1a24 P 5b818b6c74f54250b78db6a86557cddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.749328 26870 raft_consensus.cc:366] T 8d5464716606431684b2b9eec5ec1a24 P 5b818b6c74f54250b78db6a86557cddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:25.750643 26870 raft_consensus.cc:705] T 8d5464716606431684b2b9eec5ec1a24 P 5b818b6c74f54250b78db6a86557cddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b818b6c74f54250b78db6a86557cddb, State: Initialized, Role: FOLLOWER I0918 20:39:25.751421 26870 consensus_queue.cc:229] T 8d5464716606431684b2b9eec5ec1a24 P 5b818b6c74f54250b78db6a86557cddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.754596 26870 ts_tablet_manager.cc:1113] T 8d5464716606431684b2b9eec5ec1a24 P 5b818b6c74f54250b78db6a86557cddb: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s I0918 20:39:25.739195 26871 tablet_bootstrap.cc:590] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd: No blocks or log segments found. Creating new log. I0918 20:39:25.766633 26875 raft_consensus.cc:472] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:25.767128 26875 raft_consensus.cc:494] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.767956 26872 tablet_bootstrap.cc:590] T 61c6e28dc1684699a5ed860fc8a6201c P 2e5fe8bb38b645979eb1c8da1c6f5e98: No blocks or log segments found. Creating new log. I0918 20:39:25.769104 26875 leader_election.cc:296] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ed908683a4b4657aed8cbac7c7db7fd (127.18.222.2:38505), 2e5fe8bb38b645979eb1c8da1c6f5e98 (127.18.222.3:45467) I0918 20:39:25.786111 26871 tablet_bootstrap.cc:439] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd: No bootstrap required, opened a new log I0918 20:39:25.796905 26871 ts_tablet_manager.cc:1085] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.004s I0918 20:39:25.804433 26871 raft_consensus.cc:340] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.805313 26871 raft_consensus.cc:366] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:25.805570 26871 raft_consensus.cc:705] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed908683a4b4657aed8cbac7c7db7fd, State: Initialized, Role: FOLLOWER I0918 20:39:25.806267 26871 consensus_queue.cc:229] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.809193 26872 tablet_bootstrap.cc:439] T 61c6e28dc1684699a5ed860fc8a6201c P 2e5fe8bb38b645979eb1c8da1c6f5e98: No bootstrap required, opened a new log I0918 20:39:25.813407 26872 ts_tablet_manager.cc:1085] T 61c6e28dc1684699a5ed860fc8a6201c P 2e5fe8bb38b645979eb1c8da1c6f5e98: Time spent bootstrapping tablet: real 0.076s user 0.011s sys 0.009s I0918 20:39:25.813737 26871 ts_tablet_manager.cc:1113] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s I0918 20:39:25.823027 26872 raft_consensus.cc:340] T 61c6e28dc1684699a5ed860fc8a6201c P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.824301 26872 raft_consensus.cc:366] T 61c6e28dc1684699a5ed860fc8a6201c P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:25.824599 26872 raft_consensus.cc:705] T 61c6e28dc1684699a5ed860fc8a6201c P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5fe8bb38b645979eb1c8da1c6f5e98, State: Initialized, Role: FOLLOWER I0918 20:39:25.825469 26872 consensus_queue.cc:229] T 61c6e28dc1684699a5ed860fc8a6201c P 2e5fe8bb38b645979eb1c8da1c6f5e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.832646 26818 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6d805ab6e7654ea99bd1e9e6e57217cc" candidate_uuid: "5b818b6c74f54250b78db6a86557cddb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" is_pre_election: true W0918 20:39:25.836035 26629 leader_election.cc:348] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2e5fe8bb38b645979eb1c8da1c6f5e98 (127.18.222.3:45467): Illegal state: must be running to vote when last-logged opid is not known I0918 20:39:25.841351 26872 ts_tablet_manager.cc:1113] T 61c6e28dc1684699a5ed860fc8a6201c P 2e5fe8bb38b645979eb1c8da1c6f5e98: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s I0918 20:39:25.842355 26872 tablet_bootstrap.cc:439] T 6d805ab6e7654ea99bd1e9e6e57217cc P 2e5fe8bb38b645979eb1c8da1c6f5e98: Bootstrap starting. I0918 20:39:25.844413 26747 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6d805ab6e7654ea99bd1e9e6e57217cc" candidate_uuid: "5b818b6c74f54250b78db6a86557cddb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" is_pre_election: true I0918 20:39:25.845269 26747 raft_consensus.cc:2350] T 6d805ab6e7654ea99bd1e9e6e57217cc P 9ed908683a4b4657aed8cbac7c7db7fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b818b6c74f54250b78db6a86557cddb in term 0. I0918 20:39:25.845986 26872 tablet_bootstrap.cc:590] T 6d805ab6e7654ea99bd1e9e6e57217cc P 2e5fe8bb38b645979eb1c8da1c6f5e98: No blocks or log segments found. Creating new log. I0918 20:39:25.850654 26632 leader_election.cc:310] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b818b6c74f54250b78db6a86557cddb, 9ed908683a4b4657aed8cbac7c7db7fd; no voters: 2e5fe8bb38b645979eb1c8da1c6f5e98 I0918 20:39:25.852499 26875 raft_consensus.cc:2644] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:25.852900 26875 raft_consensus.cc:472] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:25.853214 26875 raft_consensus.cc:2890] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:25.872421 26875 raft_consensus.cc:494] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.875044 26872 tablet_bootstrap.cc:439] T 6d805ab6e7654ea99bd1e9e6e57217cc P 2e5fe8bb38b645979eb1c8da1c6f5e98: No bootstrap required, opened a new log I0918 20:39:25.874929 26747 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6d805ab6e7654ea99bd1e9e6e57217cc" candidate_uuid: "5b818b6c74f54250b78db6a86557cddb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" I0918 20:39:25.875581 26747 raft_consensus.cc:2890] T 6d805ab6e7654ea99bd1e9e6e57217cc P 9ed908683a4b4657aed8cbac7c7db7fd [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:25.876742 26818 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6d805ab6e7654ea99bd1e9e6e57217cc" candidate_uuid: "5b818b6c74f54250b78db6a86557cddb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" I0918 20:39:25.876580 26875 leader_election.cc:296] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [CANDIDATE]: Term 1 election: Requested vote from peers 9ed908683a4b4657aed8cbac7c7db7fd (127.18.222.2:38505), 2e5fe8bb38b645979eb1c8da1c6f5e98 (127.18.222.3:45467) W0918 20:39:25.878413 26629 leader_election.cc:348] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2e5fe8bb38b645979eb1c8da1c6f5e98 (127.18.222.3:45467): Illegal state: must be running to vote when last-logged opid is not known I0918 20:39:25.881733 26747 raft_consensus.cc:2350] T 6d805ab6e7654ea99bd1e9e6e57217cc P 9ed908683a4b4657aed8cbac7c7db7fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b818b6c74f54250b78db6a86557cddb in term 1. I0918 20:39:25.883572 26632 leader_election.cc:310] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b818b6c74f54250b78db6a86557cddb, 9ed908683a4b4657aed8cbac7c7db7fd; no voters: 2e5fe8bb38b645979eb1c8da1c6f5e98 I0918 20:39:25.885174 26872 ts_tablet_manager.cc:1085] T 6d805ab6e7654ea99bd1e9e6e57217cc P 2e5fe8bb38b645979eb1c8da1c6f5e98: Time spent bootstrapping tablet: real 0.043s user 0.018s sys 0.000s I0918 20:39:25.887276 26875 raft_consensus.cc:2644] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:25.895047 26888 raft_consensus.cc:472] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:25.896221 26875 raft_consensus.cc:667] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [term 1 LEADER]: Becoming Leader. State: Replica: 5b818b6c74f54250b78db6a86557cddb, State: Running, Role: LEADER I0918 20:39:25.896323 26888 raft_consensus.cc:494] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.895447 26872 raft_consensus.cc:340] T 6d805ab6e7654ea99bd1e9e6e57217cc P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.909549 26872 raft_consensus.cc:366] T 6d805ab6e7654ea99bd1e9e6e57217cc P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:25.909114 26875 consensus_queue.cc:206] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.909819 26872 raft_consensus.cc:705] T 6d805ab6e7654ea99bd1e9e6e57217cc P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5fe8bb38b645979eb1c8da1c6f5e98, State: Initialized, Role: FOLLOWER I0918 20:39:25.898363 26877 raft_consensus.cc:472] T 61c6e28dc1684699a5ed860fc8a6201c P 9ed908683a4b4657aed8cbac7c7db7fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:25.910544 26877 raft_consensus.cc:494] T 61c6e28dc1684699a5ed860fc8a6201c P 9ed908683a4b4657aed8cbac7c7db7fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.910692 26747 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "61c6e28dc1684699a5ed860fc8a6201c" candidate_uuid: "5b818b6c74f54250b78db6a86557cddb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" is_pre_election: true I0918 20:39:25.910670 26872 consensus_queue.cc:229] T 6d805ab6e7654ea99bd1e9e6e57217cc P 2e5fe8bb38b645979eb1c8da1c6f5e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.911926 26747 raft_consensus.cc:2350] T 61c6e28dc1684699a5ed860fc8a6201c P 9ed908683a4b4657aed8cbac7c7db7fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b818b6c74f54250b78db6a86557cddb in term 0. I0918 20:39:25.913118 26632 leader_election.cc:310] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b818b6c74f54250b78db6a86557cddb, 9ed908683a4b4657aed8cbac7c7db7fd; no voters: I0918 20:39:25.914393 26818 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "61c6e28dc1684699a5ed860fc8a6201c" candidate_uuid: "5b818b6c74f54250b78db6a86557cddb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" is_pre_election: true I0918 20:39:25.915064 26818 raft_consensus.cc:2350] T 61c6e28dc1684699a5ed860fc8a6201c P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b818b6c74f54250b78db6a86557cddb in term 0. I0918 20:39:25.915648 26888 leader_election.cc:296] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ed908683a4b4657aed8cbac7c7db7fd (127.18.222.2:38505), 2e5fe8bb38b645979eb1c8da1c6f5e98 (127.18.222.3:45467) I0918 20:39:25.916235 26888 raft_consensus.cc:2644] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:25.916525 26888 raft_consensus.cc:472] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:25.916821 26888 raft_consensus.cc:2890] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:25.920840 26877 leader_election.cc:296] T 61c6e28dc1684699a5ed860fc8a6201c P 9ed908683a4b4657aed8cbac7c7db7fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b818b6c74f54250b78db6a86557cddb (127.18.222.1:41007), 2e5fe8bb38b645979eb1c8da1c6f5e98 (127.18.222.3:45467) I0918 20:39:25.923225 26872 ts_tablet_manager.cc:1113] T 6d805ab6e7654ea99bd1e9e6e57217cc P 2e5fe8bb38b645979eb1c8da1c6f5e98: Time spent starting tablet: real 0.038s user 0.011s sys 0.007s I0918 20:39:25.946766 26888 raft_consensus.cc:494] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.956383 26747 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "61c6e28dc1684699a5ed860fc8a6201c" candidate_uuid: "5b818b6c74f54250b78db6a86557cddb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" I0918 20:39:25.957018 26747 raft_consensus.cc:2890] T 61c6e28dc1684699a5ed860fc8a6201c P 9ed908683a4b4657aed8cbac7c7db7fd [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:25.960891 26518 catalog_manager.cc:4131] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb reported cstate change: term changed from 0 to 1, leader changed from to 5b818b6c74f54250b78db6a86557cddb (127.18.222.1). New cstate: current_term: 1 leader_uuid: "5b818b6c74f54250b78db6a86557cddb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } health_report { overall_health: UNKNOWN } } } I0918 20:39:25.965016 26888 leader_election.cc:296] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [CANDIDATE]: Term 1 election: Requested vote from peers 9ed908683a4b4657aed8cbac7c7db7fd (127.18.222.2:38505), 2e5fe8bb38b645979eb1c8da1c6f5e98 (127.18.222.3:45467) I0918 20:39:25.966553 26818 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "61c6e28dc1684699a5ed860fc8a6201c" candidate_uuid: "5b818b6c74f54250b78db6a86557cddb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" I0918 20:39:25.967901 26818 raft_consensus.cc:2890] T 61c6e28dc1684699a5ed860fc8a6201c P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:25.963927 26747 raft_consensus.cc:2350] T 61c6e28dc1684699a5ed860fc8a6201c P 9ed908683a4b4657aed8cbac7c7db7fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b818b6c74f54250b78db6a86557cddb in term 1. I0918 20:39:25.970062 26632 leader_election.cc:310] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b818b6c74f54250b78db6a86557cddb, 9ed908683a4b4657aed8cbac7c7db7fd; no voters: I0918 20:39:25.972379 26875 raft_consensus.cc:2644] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:25.973412 26875 raft_consensus.cc:667] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [term 1 LEADER]: Becoming Leader. State: Replica: 5b818b6c74f54250b78db6a86557cddb, State: Running, Role: LEADER I0918 20:39:25.974139 26875 consensus_queue.cc:206] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:25.988059 26676 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "61c6e28dc1684699a5ed860fc8a6201c" candidate_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b818b6c74f54250b78db6a86557cddb" is_pre_election: true I0918 20:39:25.991744 26676 raft_consensus.cc:2311] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 9ed908683a4b4657aed8cbac7c7db7fd for term 1 because replica is either leader or believes a valid leader to be alive. I0918 20:39:25.985393 26818 raft_consensus.cc:2350] T 61c6e28dc1684699a5ed860fc8a6201c P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b818b6c74f54250b78db6a86557cddb in term 1. I0918 20:39:26.011838 26877 raft_consensus.cc:472] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:26.012351 26877 raft_consensus.cc:494] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:26.014119 26877 leader_election.cc:296] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b818b6c74f54250b78db6a86557cddb (127.18.222.1:41007), 2e5fe8bb38b645979eb1c8da1c6f5e98 (127.18.222.3:45467) I0918 20:39:26.023097 26676 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8d5464716606431684b2b9eec5ec1a24" candidate_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b818b6c74f54250b78db6a86557cddb" is_pre_election: true I0918 20:39:26.025688 26676 raft_consensus.cc:2350] T 8d5464716606431684b2b9eec5ec1a24 P 5b818b6c74f54250b78db6a86557cddb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed908683a4b4657aed8cbac7c7db7fd in term 0. I0918 20:39:26.030614 26708 leader_election.cc:310] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b818b6c74f54250b78db6a86557cddb, 9ed908683a4b4657aed8cbac7c7db7fd; no voters: I0918 20:39:26.032462 26877 raft_consensus.cc:2644] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:26.032833 26877 raft_consensus.cc:472] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:26.033123 26877 raft_consensus.cc:2890] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:26.050478 26877 raft_consensus.cc:494] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:26.052145 26877 leader_election.cc:296] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd [CANDIDATE]: Term 1 election: Requested vote from peers 5b818b6c74f54250b78db6a86557cddb (127.18.222.1:41007), 2e5fe8bb38b645979eb1c8da1c6f5e98 (127.18.222.3:45467) I0918 20:39:26.053047 26676 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8d5464716606431684b2b9eec5ec1a24" candidate_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b818b6c74f54250b78db6a86557cddb" I0918 20:39:26.053652 26676 raft_consensus.cc:2890] T 8d5464716606431684b2b9eec5ec1a24 P 5b818b6c74f54250b78db6a86557cddb [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:26.062551 26676 raft_consensus.cc:2350] T 8d5464716606431684b2b9eec5ec1a24 P 5b818b6c74f54250b78db6a86557cddb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed908683a4b4657aed8cbac7c7db7fd in term 1. I0918 20:39:26.063560 26708 leader_election.cc:310] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b818b6c74f54250b78db6a86557cddb, 9ed908683a4b4657aed8cbac7c7db7fd; no voters: I0918 20:39:26.069272 26877 raft_consensus.cc:2644] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:26.070726 26517 catalog_manager.cc:4131] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb reported cstate change: term changed from 0 to 1, leader changed from to 5b818b6c74f54250b78db6a86557cddb (127.18.222.1). New cstate: current_term: 1 leader_uuid: "5b818b6c74f54250b78db6a86557cddb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } health_report { overall_health: UNKNOWN } } } I0918 20:39:26.074137 26818 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "61c6e28dc1684699a5ed860fc8a6201c" candidate_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" is_pre_election: true I0918 20:39:26.074632 26817 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8d5464716606431684b2b9eec5ec1a24" candidate_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" is_pre_election: true I0918 20:39:26.075146 26818 raft_consensus.cc:2280] T 61c6e28dc1684699a5ed860fc8a6201c P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ed908683a4b4657aed8cbac7c7db7fd in current term 1: Already voted for candidate 5b818b6c74f54250b78db6a86557cddb in this term. I0918 20:39:26.075412 26817 raft_consensus.cc:2350] T 8d5464716606431684b2b9eec5ec1a24 P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed908683a4b4657aed8cbac7c7db7fd in term 0. I0918 20:39:26.076447 26708 leader_election.cc:310] T 61c6e28dc1684699a5ed860fc8a6201c P 9ed908683a4b4657aed8cbac7c7db7fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ed908683a4b4657aed8cbac7c7db7fd; no voters: 2e5fe8bb38b645979eb1c8da1c6f5e98, 5b818b6c74f54250b78db6a86557cddb I0918 20:39:26.077117 26817 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8d5464716606431684b2b9eec5ec1a24" candidate_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" I0918 20:39:26.078064 26817 raft_consensus.cc:2890] T 8d5464716606431684b2b9eec5ec1a24 P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:26.087280 26817 raft_consensus.cc:2350] T 8d5464716606431684b2b9eec5ec1a24 P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed908683a4b4657aed8cbac7c7db7fd in term 1. I0918 20:39:26.109405 26877 raft_consensus.cc:667] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd [term 1 LEADER]: Becoming Leader. State: Replica: 9ed908683a4b4657aed8cbac7c7db7fd, State: Running, Role: LEADER I0918 20:39:26.110711 26877 consensus_queue.cc:206] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:26.119055 26902 raft_consensus.cc:2592] T 61c6e28dc1684699a5ed860fc8a6201c P 9ed908683a4b4657aed8cbac7c7db7fd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0918 20:39:26.127430 26517 catalog_manager.cc:4131] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd reported cstate change: term changed from 0 to 1, leader changed from to 9ed908683a4b4657aed8cbac7c7db7fd (127.18.222.2). New cstate: current_term: 1 leader_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } health_report { overall_health: UNKNOWN } } } I0918 20:39:26.155467 26879 raft_consensus.cc:472] T 6d805ab6e7654ea99bd1e9e6e57217cc P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:26.155932 26879 raft_consensus.cc:494] T 6d805ab6e7654ea99bd1e9e6e57217cc P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 } } peers { permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 } } peers { permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 } } I0918 20:39:26.157541 26879 leader_election.cc:296] T 6d805ab6e7654ea99bd1e9e6e57217cc P 2e5fe8bb38b645979eb1c8da1c6f5e98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b818b6c74f54250b78db6a86557cddb (127.18.222.1:41007), 9ed908683a4b4657aed8cbac7c7db7fd (127.18.222.2:38505) I0918 20:39:26.192683 26676 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6d805ab6e7654ea99bd1e9e6e57217cc" candidate_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b818b6c74f54250b78db6a86557cddb" is_pre_election: true I0918 20:39:26.193558 26676 raft_consensus.cc:2311] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 2e5fe8bb38b645979eb1c8da1c6f5e98 for term 1 because replica is either leader or believes a valid leader to be alive. I0918 20:39:26.198334 26747 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6d805ab6e7654ea99bd1e9e6e57217cc" candidate_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" is_pre_election: true I0918 20:39:26.199172 26747 raft_consensus.cc:2280] T 6d805ab6e7654ea99bd1e9e6e57217cc P 9ed908683a4b4657aed8cbac7c7db7fd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2e5fe8bb38b645979eb1c8da1c6f5e98 in current term 1: Already voted for candidate 5b818b6c74f54250b78db6a86557cddb in this term. I0918 20:39:26.201134 26782 leader_election.cc:310] T 6d805ab6e7654ea99bd1e9e6e57217cc P 2e5fe8bb38b645979eb1c8da1c6f5e98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e5fe8bb38b645979eb1c8da1c6f5e98; no voters: 5b818b6c74f54250b78db6a86557cddb, 9ed908683a4b4657aed8cbac7c7db7fd I0918 20:39:26.202034 26879 raft_consensus.cc:2890] T 6d805ab6e7654ea99bd1e9e6e57217cc P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:26.207455 19320 ksck_remote-test.cc:693] Generating row writes... I0918 20:39:26.210465 26879 raft_consensus.cc:2592] T 6d805ab6e7654ea99bd1e9e6e57217cc P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0918 20:39:26.367076 26817 raft_consensus.cc:1184] T 6d805ab6e7654ea99bd1e9e6e57217cc P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 1 FOLLOWER]: Refusing update from remote peer 5b818b6c74f54250b78db6a86557cddb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:26.372226 26747 raft_consensus.cc:1184] T 6d805ab6e7654ea99bd1e9e6e57217cc P 9ed908683a4b4657aed8cbac7c7db7fd [term 1 FOLLOWER]: Refusing update from remote peer 5b818b6c74f54250b78db6a86557cddb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:26.374974 26875 consensus_queue.cc:984] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:26.377319 26888 consensus_queue.cc:984] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:26.379335 26817 raft_consensus.cc:1184] T 8d5464716606431684b2b9eec5ec1a24 P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 1 FOLLOWER]: Refusing update from remote peer 9ed908683a4b4657aed8cbac7c7db7fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:26.381189 26676 raft_consensus.cc:1184] T 8d5464716606431684b2b9eec5ec1a24 P 5b818b6c74f54250b78db6a86557cddb [term 1 FOLLOWER]: Refusing update from remote peer 9ed908683a4b4657aed8cbac7c7db7fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:26.384783 26877 consensus_queue.cc:984] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b818b6c74f54250b78db6a86557cddb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:26.386122 26877 consensus_queue.cc:984] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:26.437866 26746 raft_consensus.cc:1184] T 61c6e28dc1684699a5ed860fc8a6201c P 9ed908683a4b4657aed8cbac7c7db7fd [term 1 FOLLOWER]: Refusing update from remote peer 5b818b6c74f54250b78db6a86557cddb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:26.441277 26875 consensus_queue.cc:984] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed908683a4b4657aed8cbac7c7db7fd" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:26.448616 26816 raft_consensus.cc:1184] T 61c6e28dc1684699a5ed860fc8a6201c P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 1 FOLLOWER]: Refusing update from remote peer 5b818b6c74f54250b78db6a86557cddb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:26.452662 26875 consensus_queue.cc:984] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e5fe8bb38b645979eb1c8da1c6f5e98" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 45467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:27.669806 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:27.742494 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:27.742825 19320 tablet_replica.cc:273] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb: stopping tablet replica I0918 20:39:27.743294 19320 raft_consensus.cc:2147] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:27.744272 19320 raft_consensus.cc:2174] T 61c6e28dc1684699a5ed860fc8a6201c P 5b818b6c74f54250b78db6a86557cddb [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:27.747092 19320 tablet_replica.cc:273] T 8d5464716606431684b2b9eec5ec1a24 P 5b818b6c74f54250b78db6a86557cddb: stopping tablet replica I0918 20:39:27.747470 19320 raft_consensus.cc:2147] T 8d5464716606431684b2b9eec5ec1a24 P 5b818b6c74f54250b78db6a86557cddb [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:27.747928 19320 raft_consensus.cc:2174] T 8d5464716606431684b2b9eec5ec1a24 P 5b818b6c74f54250b78db6a86557cddb [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:27.750273 19320 tablet_replica.cc:273] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb: stopping tablet replica I0918 20:39:27.750705 19320 raft_consensus.cc:2147] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:27.751399 19320 raft_consensus.cc:2174] T 6d805ab6e7654ea99bd1e9e6e57217cc P 5b818b6c74f54250b78db6a86557cddb [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:27.769270 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:27.769551 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:27.770022 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:27.786254 19320 tablet_server.cc:155] TabletServer shutting down... W0918 20:39:27.828689 26708 consensus_peers.cc:458] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd -> Peer 5b818b6c74f54250b78db6a86557cddb (127.18.222.1:41007): Couldn't send request to peer 5b818b6c74f54250b78db6a86557cddb. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.1:41007: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:39:27.839941 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:27.840273 19320 tablet_replica.cc:273] T 61c6e28dc1684699a5ed860fc8a6201c P 9ed908683a4b4657aed8cbac7c7db7fd: stopping tablet replica I0918 20:39:27.840684 19320 raft_consensus.cc:2147] T 61c6e28dc1684699a5ed860fc8a6201c P 9ed908683a4b4657aed8cbac7c7db7fd [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:27.841141 19320 raft_consensus.cc:2174] T 61c6e28dc1684699a5ed860fc8a6201c P 9ed908683a4b4657aed8cbac7c7db7fd [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:27.843503 19320 tablet_replica.cc:273] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd: stopping tablet replica I0918 20:39:27.843902 19320 raft_consensus.cc:2147] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:27.844641 19320 raft_consensus.cc:2174] T 8d5464716606431684b2b9eec5ec1a24 P 9ed908683a4b4657aed8cbac7c7db7fd [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:27.846962 19320 tablet_replica.cc:273] T 6d805ab6e7654ea99bd1e9e6e57217cc P 9ed908683a4b4657aed8cbac7c7db7fd: stopping tablet replica I0918 20:39:27.847290 19320 raft_consensus.cc:2147] T 6d805ab6e7654ea99bd1e9e6e57217cc P 9ed908683a4b4657aed8cbac7c7db7fd [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:27.847553 19320 raft_consensus.cc:2174] T 6d805ab6e7654ea99bd1e9e6e57217cc P 9ed908683a4b4657aed8cbac7c7db7fd [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:27.861915 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:27.862146 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:27.862454 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:27.875294 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:27.907269 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:27.907582 19320 tablet_replica.cc:273] T 61c6e28dc1684699a5ed860fc8a6201c P 2e5fe8bb38b645979eb1c8da1c6f5e98: stopping tablet replica I0918 20:39:27.908004 19320 raft_consensus.cc:2147] T 61c6e28dc1684699a5ed860fc8a6201c P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:27.908303 19320 raft_consensus.cc:2174] T 61c6e28dc1684699a5ed860fc8a6201c P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:27.910568 19320 tablet_replica.cc:273] T 6d805ab6e7654ea99bd1e9e6e57217cc P 2e5fe8bb38b645979eb1c8da1c6f5e98: stopping tablet replica I0918 20:39:27.910950 19320 raft_consensus.cc:2147] T 6d805ab6e7654ea99bd1e9e6e57217cc P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:27.911214 19320 raft_consensus.cc:2174] T 6d805ab6e7654ea99bd1e9e6e57217cc P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:27.913120 19320 tablet_replica.cc:273] T 8d5464716606431684b2b9eec5ec1a24 P 2e5fe8bb38b645979eb1c8da1c6f5e98: stopping tablet replica I0918 20:39:27.913437 19320 raft_consensus.cc:2147] T 8d5464716606431684b2b9eec5ec1a24 P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:27.913704 19320 raft_consensus.cc:2174] T 8d5464716606431684b2b9eec5ec1a24 P 2e5fe8bb38b645979eb1c8da1c6f5e98 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:27.934252 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:27.934449 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:27.934844 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:27.945329 19320 master.cc:247] Master@127.18.222.62:38661 shutting down... I0918 20:39:27.963517 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:27.963974 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:27.964257 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 8093d7d8c08b46af9003e7ae02eea403: stopping tablet replica I0918 20:39:27.982316 19320 master.cc:258] Master@127.18.222.62:38661 shutdown complete. I0918 20:39:27.991138 19320 master.cc:247] Master@127.18.222.61:43053 shutting down... I0918 20:39:28.010910 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:28.011755 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:28.012027 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P b365daeaabca4390bae27ba6253826cf: stopping tablet replica I0918 20:39:28.031002 19320 master.cc:258] Master@127.18.222.61:43053 shutdown complete. I0918 20:39:28.041575 19320 master.cc:247] Master@127.18.222.60:45059 shutting down... I0918 20:39:28.060705 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:28.061075 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:28.061285 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 2a4885def8064583a05ff501f6773dc8: stopping tablet replica I0918 20:39:28.079355 19320 master.cc:258] Master@127.18.222.60:45059 shutdown complete. [ OK ] RemoteKsckTest.TestMatchingTableFilter (3930 ms) [ RUN ] RemoteKsckTest.TestFilterOnNotabletTable I0918 20:39:28.111603 19320 test_util.cc:212] Using random seed: -715967531 I0918 20:39:28.112418 19320 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.18.222.62:44123,127.18.222.61:40549,127.18.222.60:32967 I0918 20:39:28.125128 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 161592us I0918 20:39:28.125481 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:28.125622 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:28.125810 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:28.125921 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:28.126914 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/master-0-root/instance: uuid: "065d5720fcff45859a00631da55c7ef9" format_stamp: "Formatted at 2019-09-18 20:39:28 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:28.135957 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.010s sys 0.001s I0918 20:39:28.137449 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:28.144093 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:28.144263 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/master-0-root uuid: "065d5720fcff45859a00631da55c7ef9" format_stamp: "Formatted at 2019-09-18 20:39:28 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:28.144508 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:28.210283 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:44123 I0918 20:39:28.212028 19320 webserver.cc:374] Webserver started at http://127.18.222.62:33125/ using document root and password file I0918 20:39:28.213809 27022 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:28.218647 27022 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:28.230293 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 161592us I0918 20:39:28.230839 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:28.231045 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:28.231333 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:28.231510 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:28.232977 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/master-1-root/instance: uuid: "a35f07dedf474aab830daea0ff904099" format_stamp: "Formatted at 2019-09-18 20:39:28 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:28.236410 27022 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:28.238168 27022 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:40549: Network error: Client connection negotiation failed: client connection to 127.18.222.61:40549: connect: Connection refused (error 111) I0918 20:39:28.244318 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.012s sys 0.000s I0918 20:39:28.246111 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:28.253795 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:28.253974 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/master-1-root uuid: "a35f07dedf474aab830daea0ff904099" format_stamp: "Formatted at 2019-09-18 20:39:28 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:28.254243 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:28.277546 27022 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } attempt: 1 W0918 20:39:28.279917 27022 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:40549: Network error: Client connection negotiation failed: client connection to 127.18.222.61:40549: connect: Connection refused (error 111) I0918 20:39:28.307026 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:40549 I0918 20:39:28.308888 19320 webserver.cc:374] Webserver started at http://127.18.222.61:43797/ using document root and password file I0918 20:39:28.310681 27079 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:28.315503 27079 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:28.328075 27079 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:28.328222 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 161592us I0918 20:39:28.328611 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:28.328814 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:28.329088 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:28.329272 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:28.330551 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/master-2-root/instance: uuid: "3514894dbf0a484baa971e988b96db25" format_stamp: "Formatted at 2019-09-18 20:39:28 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:28.341686 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.013s sys 0.000s I0918 20:39:28.343921 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:28.346869 27079 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:28.348385 27022 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } attempt: 2 W0918 20:39:28.348809 27079 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:32967: Network error: Client connection negotiation failed: client connection to 127.18.222.60:32967: connect: Connection refused (error 111) I0918 20:39:28.355352 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0918 20:39:28.355789 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/master-2-root uuid: "3514894dbf0a484baa971e988b96db25" format_stamp: "Formatted at 2019-09-18 20:39:28 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:28.356151 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:28.360422 27022 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:28.362416 27022 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:32967: Network error: Client connection negotiation failed: client connection to 127.18.222.60:32967: connect: Connection refused (error 111) I0918 20:39:28.379263 27079 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } attempt: 1 W0918 20:39:28.381398 27079 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:32967: Network error: Client connection negotiation failed: client connection to 127.18.222.60:32967: connect: Connection refused (error 111) I0918 20:39:28.384862 27022 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } attempt: 1 W0918 20:39:28.386914 27022 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:32967: Network error: Client connection negotiation failed: client connection to 127.18.222.60:32967: connect: Connection refused (error 111) I0918 20:39:28.425909 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:32967 I0918 20:39:28.428042 19320 webserver.cc:374] Webserver started at http://127.18.222.60:44857/ using document root and password file I0918 20:39:28.429215 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0918 20:39:28.430116 27136 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:28.435397 27136 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:28.442804 27079 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } attempt: 2 I0918 20:39:28.445369 27022 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } attempt: 2 I0918 20:39:28.450570 27136 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:28.466550 27136 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:28.469739 27079 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099: Bootstrap starting. I0918 20:39:28.472728 27079 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099: No blocks or log segments found. Creating new log. I0918 20:39:28.474279 27022 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9: Bootstrap starting. I0918 20:39:28.478288 27022 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9: No blocks or log segments found. Creating new log. I0918 20:39:28.480628 27079 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099: No bootstrap required, opened a new log I0918 20:39:28.481863 27022 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9: No bootstrap required, opened a new log I0918 20:39:28.483049 27079 raft_consensus.cc:340] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } I0918 20:39:28.483599 27079 raft_consensus.cc:366] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:28.483867 27079 raft_consensus.cc:705] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a35f07dedf474aab830daea0ff904099, State: Initialized, Role: FOLLOWER I0918 20:39:28.484421 27079 consensus_queue.cc:229] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } I0918 20:39:28.484197 27022 raft_consensus.cc:340] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } I0918 20:39:28.485088 27022 raft_consensus.cc:366] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:28.485337 27022 raft_consensus.cc:705] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 065d5720fcff45859a00631da55c7ef9, State: Initialized, Role: FOLLOWER I0918 20:39:28.485951 27022 consensus_queue.cc:229] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } I0918 20:39:28.490072 27136 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25: Bootstrap starting. I0918 20:39:28.492439 27079 sys_catalog.cc:340] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } } I0918 20:39:28.493094 27079 sys_catalog.cc:343] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:28.492758 27142 sys_catalog.cc:340] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } } I0918 20:39:28.493712 27142 sys_catalog.cc:343] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:28.494117 27079 sys_catalog.cc:428] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:28.494863 27022 sys_catalog.cc:340] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } } I0918 20:39:28.495750 27022 sys_catalog.cc:343] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:28.495272 27143 sys_catalog.cc:340] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } } I0918 20:39:28.496688 27143 sys_catalog.cc:343] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:28.497196 27022 sys_catalog.cc:428] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:28.498498 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 I0918 20:39:28.498531 27136 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25: No blocks or log segments found. Creating new log. I0918 20:39:28.498800 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 W0918 20:39:28.499593 27144 catalog_manager.cc:1155] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:28.499996 27144 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0918 20:39:28.500728 27145 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:28.500986 27145 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:28.502465 27136 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25: No bootstrap required, opened a new log I0918 20:39:28.504389 27136 raft_consensus.cc:340] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } I0918 20:39:28.504874 27136 raft_consensus.cc:366] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:28.505071 27136 raft_consensus.cc:705] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3514894dbf0a484baa971e988b96db25, State: Initialized, Role: FOLLOWER I0918 20:39:28.505568 27136 consensus_queue.cc:229] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } I0918 20:39:28.507012 27136 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } } I0918 20:39:28.507459 27136 sys_catalog.cc:343] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:28.507192 27147 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } } I0918 20:39:28.507941 27147 sys_catalog.cc:343] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:28.508455 27136 sys_catalog.cc:428] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [sys.catalog]: configured and running, proceeding with master startup. W0918 20:39:28.511495 27148 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:28.511723 27148 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:28.525138 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 161592us I0918 20:39:28.525537 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:28.525707 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:28.525933 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:28.526065 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:28.527112 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/ts-0-root/instance: uuid: "2d9779d348c74e069521dc32fa90746e" format_stamp: "Formatted at 2019-09-18 20:39:28 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:28.532121 27147 raft_consensus.cc:472] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:28.532719 27147 raft_consensus.cc:494] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } I0918 20:39:28.534549 27147 leader_election.cc:296] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 065d5720fcff45859a00631da55c7ef9 (127.18.222.62:44123), a35f07dedf474aab830daea0ff904099 (127.18.222.61:40549) I0918 20:39:28.535527 27056 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3514894dbf0a484baa971e988b96db25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35f07dedf474aab830daea0ff904099" is_pre_election: true I0918 20:39:28.535465 26999 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3514894dbf0a484baa971e988b96db25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "065d5720fcff45859a00631da55c7ef9" is_pre_election: true I0918 20:39:28.536350 27056 raft_consensus.cc:2350] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3514894dbf0a484baa971e988b96db25 in term 0. I0918 20:39:28.536417 26999 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3514894dbf0a484baa971e988b96db25 in term 0. I0918 20:39:28.537310 27084 leader_election.cc:310] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3514894dbf0a484baa971e988b96db25, a35f07dedf474aab830daea0ff904099; no voters: I0918 20:39:28.538146 27147 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:28.538455 27147 raft_consensus.cc:472] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:28.538880 27147 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:28.539633 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.012s user 0.007s sys 0.008s I0918 20:39:28.542110 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:28.544983 27147 raft_consensus.cc:494] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } I0918 20:39:28.546746 27147 leader_election.cc:296] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [CANDIDATE]: Term 1 election: Requested vote from peers 065d5720fcff45859a00631da55c7ef9 (127.18.222.62:44123), a35f07dedf474aab830daea0ff904099 (127.18.222.61:40549) I0918 20:39:28.547607 26999 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3514894dbf0a484baa971e988b96db25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "065d5720fcff45859a00631da55c7ef9" I0918 20:39:28.548271 26999 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:28.548068 27056 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3514894dbf0a484baa971e988b96db25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35f07dedf474aab830daea0ff904099" I0918 20:39:28.549041 27056 raft_consensus.cc:2890] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:28.553097 27056 raft_consensus.cc:2350] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3514894dbf0a484baa971e988b96db25 in term 1. I0918 20:39:28.554085 27084 leader_election.cc:310] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3514894dbf0a484baa971e988b96db25, a35f07dedf474aab830daea0ff904099; no voters: I0918 20:39:28.554244 26999 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3514894dbf0a484baa971e988b96db25 in term 1. I0918 20:39:28.554854 27147 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:28.556485 27147 raft_consensus.cc:667] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [term 1 LEADER]: Becoming Leader. State: Replica: 3514894dbf0a484baa971e988b96db25, State: Running, Role: LEADER I0918 20:39:28.556792 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0918 20:39:28.557082 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/ts-0-root uuid: "2d9779d348c74e069521dc32fa90746e" format_stamp: "Formatted at 2019-09-18 20:39:28 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:28.557379 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:28.557112 27147 consensus_queue.cc:206] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } I0918 20:39:28.561158 27151 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3514894dbf0a484baa971e988b96db25. Latest consensus state: current_term: 1 leader_uuid: "3514894dbf0a484baa971e988b96db25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } } I0918 20:39:28.561816 27151 sys_catalog.cc:343] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [sys.catalog]: This master's current role is: LEADER I0918 20:39:28.564229 27155 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:39:28.566857 27155 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:39:28.588706 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:28.588909 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:28.609889 27056 raft_consensus.cc:1184] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099 [term 1 FOLLOWER]: Refusing update from remote peer 3514894dbf0a484baa971e988b96db25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:28.609917 26999 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9 [term 1 FOLLOWER]: Refusing update from remote peer 3514894dbf0a484baa971e988b96db25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:28.611647 27147 consensus_queue.cc:984] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:39:28.612695 27151 consensus_queue.cc:984] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:28.621385 27143 sys_catalog.cc:340] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3514894dbf0a484baa971e988b96db25. Latest consensus state: current_term: 1 leader_uuid: "3514894dbf0a484baa971e988b96db25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } } I0918 20:39:28.623098 27143 sys_catalog.cc:343] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:28.624610 27142 sys_catalog.cc:340] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3514894dbf0a484baa971e988b96db25. Latest consensus state: current_term: 1 leader_uuid: "3514894dbf0a484baa971e988b96db25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } } I0918 20:39:28.625385 27142 sys_catalog.cc:343] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:28.627779 27167 mvcc.cc:203] Tried to move safe_time back from 6425965234611421184 to 6425965234416275456. Current Snapshot: MvccSnapshot[committed={T|T < 6425965234611421184}] I0918 20:39:28.639449 27142 sys_catalog.cc:340] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3514894dbf0a484baa971e988b96db25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } } I0918 20:39:28.641131 27151 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3514894dbf0a484baa971e988b96db25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } } I0918 20:39:28.642055 27151 sys_catalog.cc:343] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [sys.catalog]: This master's current role is: LEADER I0918 20:39:28.642168 27142 sys_catalog.cc:343] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:28.641265 27143 sys_catalog.cc:340] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3514894dbf0a484baa971e988b96db25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } } I0918 20:39:28.643607 27143 sys_catalog.cc:343] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:28.650072 27155 catalog_manager.cc:981] Generated new certificate authority record I0918 20:39:28.653934 27155 catalog_manager.cc:1106] Loading token signing keys... I0918 20:39:28.650063 27191 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3514894dbf0a484baa971e988b96db25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065d5720fcff45859a00631da55c7ef9" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44123 } } peers { permanent_uuid: "a35f07dedf474aab830daea0ff904099" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40549 } } peers { permanent_uuid: "3514894dbf0a484baa971e988b96db25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 32967 } } } I0918 20:39:28.660791 27191 sys_catalog.cc:343] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [sys.catalog]: This master's current role is: LEADER I0918 20:39:28.679183 27155 catalog_manager.cc:4446] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25: Generated new TSK 0 I0918 20:39:28.680830 27155 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:39:28.687911 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:41505 I0918 20:39:28.689827 19320 webserver.cc:374] Webserver started at http://127.18.222.1:38889/ using document root and password file I0918 20:39:28.710853 27226 heartbeater.cc:322] Connected to a master server at 127.18.222.60:32967 I0918 20:39:28.711251 27226 heartbeater.cc:406] Registering TS with master... I0918 20:39:28.711864 27226 heartbeater.cc:455] Master 127.18.222.60:32967 requested a full tablet report, sending... I0918 20:39:28.713958 27103 ts_manager.cc:186] Registered new tserver with Master: 2d9779d348c74e069521dc32fa90746e (127.18.222.1:41505) I0918 20:39:28.715502 27227 heartbeater.cc:322] Connected to a master server at 127.18.222.61:40549 I0918 20:39:28.715998 27227 heartbeater.cc:406] Registering TS with master... I0918 20:39:28.716580 27227 heartbeater.cc:455] Master 127.18.222.61:40549 requested a full tablet report, sending... I0918 20:39:28.717046 27103 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39304 I0918 20:39:28.719121 27046 ts_manager.cc:186] Registered new tserver with Master: 2d9779d348c74e069521dc32fa90746e (127.18.222.1:41505) I0918 20:39:28.723042 27228 heartbeater.cc:322] Connected to a master server at 127.18.222.62:44123 I0918 20:39:28.723278 27228 heartbeater.cc:406] Registering TS with master... I0918 20:39:28.723618 27228 heartbeater.cc:455] Master 127.18.222.62:44123 requested a full tablet report, sending... I0918 20:39:28.725525 26989 ts_manager.cc:186] Registered new tserver with Master: 2d9779d348c74e069521dc32fa90746e (127.18.222.1:41505) I0918 20:39:28.730942 27226 heartbeater.cc:447] Master 127.18.222.60:32967 was elected leader, sending a full tablet report... I0918 20:39:28.732254 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 161592us I0918 20:39:28.732697 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:28.732883 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:28.733111 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:28.733256 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:28.734603 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/ts-1-root/instance: uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" format_stamp: "Formatted at 2019-09-18 20:39:28 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:28.746007 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.007s sys 0.004s I0918 20:39:28.748160 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:28.758000 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:28.758298 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/ts-1-root uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" format_stamp: "Formatted at 2019-09-18 20:39:28 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:28.758581 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:28.818526 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:28.818727 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:28.871500 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.2:43213 I0918 20:39:28.874055 19320 webserver.cc:374] Webserver started at http://127.18.222.2:41821/ using document root and password file I0918 20:39:28.910353 27297 heartbeater.cc:322] Connected to a master server at 127.18.222.60:32967 I0918 20:39:28.910738 27297 heartbeater.cc:406] Registering TS with master... I0918 20:39:28.911181 27297 heartbeater.cc:455] Master 127.18.222.60:32967 requested a full tablet report, sending... I0918 20:39:28.913192 27103 ts_manager.cc:186] Registered new tserver with Master: df189ce5bd184e2dac9a57ccb83e6cf9 (127.18.222.2:43213) I0918 20:39:28.915515 27103 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39310 I0918 20:39:28.922493 27298 heartbeater.cc:322] Connected to a master server at 127.18.222.61:40549 I0918 20:39:28.922972 27298 heartbeater.cc:406] Registering TS with master... I0918 20:39:28.923403 27298 heartbeater.cc:455] Master 127.18.222.61:40549 requested a full tablet report, sending... I0918 20:39:28.924021 27299 heartbeater.cc:322] Connected to a master server at 127.18.222.62:44123 I0918 20:39:28.924439 27299 heartbeater.cc:406] Registering TS with master... I0918 20:39:28.924837 27299 heartbeater.cc:455] Master 127.18.222.62:44123 requested a full tablet report, sending... I0918 20:39:28.925433 27046 ts_manager.cc:186] Registered new tserver with Master: df189ce5bd184e2dac9a57ccb83e6cf9 (127.18.222.2:43213) I0918 20:39:28.927160 26989 ts_manager.cc:186] Registered new tserver with Master: df189ce5bd184e2dac9a57ccb83e6cf9 (127.18.222.2:43213) I0918 20:39:28.929941 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 161592us I0918 20:39:28.930400 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:28.930630 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:28.930923 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:28.931095 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:28.931663 27297 heartbeater.cc:447] Master 127.18.222.60:32967 was elected leader, sending a full tablet report... I0918 20:39:28.932165 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/ts-2-root/instance: uuid: "56bdddb1aabd482b86ac1fdc71e75369" format_stamp: "Formatted at 2019-09-18 20:39:28 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:28.946017 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.013s user 0.009s sys 0.004s I0918 20:39:28.948079 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:28.959264 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:39:28.959589 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/ts-2-root uuid: "56bdddb1aabd482b86ac1fdc71e75369" format_stamp: "Formatted at 2019-09-18 20:39:28 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:28.959961 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:28.991580 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:28.991907 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:29.056972 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.3:34229 I0918 20:39:29.060881 19320 webserver.cc:374] Webserver started at http://127.18.222.3:42889/ using document root and password file I0918 20:39:29.104638 27369 heartbeater.cc:322] Connected to a master server at 127.18.222.61:40549 I0918 20:39:29.105005 27369 heartbeater.cc:406] Registering TS with master... I0918 20:39:29.105607 27369 heartbeater.cc:455] Master 127.18.222.61:40549 requested a full tablet report, sending... I0918 20:39:29.108374 27046 ts_manager.cc:186] Registered new tserver with Master: 56bdddb1aabd482b86ac1fdc71e75369 (127.18.222.3:34229) I0918 20:39:29.116148 27368 heartbeater.cc:322] Connected to a master server at 127.18.222.60:32967 I0918 20:39:29.116544 27368 heartbeater.cc:406] Registering TS with master... I0918 20:39:29.117184 27368 heartbeater.cc:455] Master 127.18.222.60:32967 requested a full tablet report, sending... I0918 20:39:29.119361 27101 ts_manager.cc:186] Registered new tserver with Master: 56bdddb1aabd482b86ac1fdc71e75369 (127.18.222.3:34229) I0918 20:39:29.119866 27372 heartbeater.cc:322] Connected to a master server at 127.18.222.62:44123 I0918 20:39:29.120206 27372 heartbeater.cc:406] Registering TS with master... I0918 20:39:29.120760 27372 heartbeater.cc:455] Master 127.18.222.62:44123 requested a full tablet report, sending... I0918 20:39:29.121414 27101 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39316 I0918 20:39:29.123268 26989 ts_manager.cc:186] Registered new tserver with Master: 56bdddb1aabd482b86ac1fdc71e75369 (127.18.222.3:34229) I0918 20:39:29.126511 19320 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.026479518s I0918 20:39:29.135213 27368 heartbeater.cc:447] Master 127.18.222.60:32967 was elected leader, sending a full tablet report... I0918 20:39:29.204306 27101 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39322: name: "ksck-test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\n\000\000\000\004\001\024\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:39:29.205974 27101 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:39:29.399924 27193 tablet_service.cc:1022] Processing CreateTablet for tablet 2267a602306e4f248b73b62cf32bc6f3 (table=ksck-test-table [id=15622109a20a45faa9a893f6023b1c6c]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:29.401018 27193 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 2267a602306e4f248b73b62cf32bc6f3. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:29.412575 27189 tablet_service.cc:1022] Processing CreateTablet for tablet 3e240c7a9d654cac8b05288625596c49 (table=ksck-test-table [id=15622109a20a45faa9a893f6023b1c6c]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:29.413853 27189 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 3e240c7a9d654cac8b05288625596c49. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:29.412849 27186 tablet_service.cc:1022] Processing CreateTablet for tablet 76338445b4aa4c5abd99265bc7f96971 (table=ksck-test-table [id=15622109a20a45faa9a893f6023b1c6c]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:29.416891 27186 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 76338445b4aa4c5abd99265bc7f96971. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:29.439283 27264 tablet_service.cc:1022] Processing CreateTablet for tablet 2267a602306e4f248b73b62cf32bc6f3 (table=ksck-test-table [id=15622109a20a45faa9a893f6023b1c6c]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:29.440559 27264 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 2267a602306e4f248b73b62cf32bc6f3. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:29.440359 27334 tablet_service.cc:1022] Processing CreateTablet for tablet 3e240c7a9d654cac8b05288625596c49 (table=ksck-test-table [id=15622109a20a45faa9a893f6023b1c6c]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:29.441377 27334 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 3e240c7a9d654cac8b05288625596c49. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:29.443063 27333 tablet_service.cc:1022] Processing CreateTablet for tablet 76338445b4aa4c5abd99265bc7f96971 (table=ksck-test-table [id=15622109a20a45faa9a893f6023b1c6c]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:29.439584 27335 tablet_service.cc:1022] Processing CreateTablet for tablet 2267a602306e4f248b73b62cf32bc6f3 (table=ksck-test-table [id=15622109a20a45faa9a893f6023b1c6c]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:29.444178 27333 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 76338445b4aa4c5abd99265bc7f96971. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:29.446200 27263 tablet_service.cc:1022] Processing CreateTablet for tablet 3e240c7a9d654cac8b05288625596c49 (table=ksck-test-table [id=15622109a20a45faa9a893f6023b1c6c]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:29.447957 27263 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 3e240c7a9d654cac8b05288625596c49. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:29.448166 27335 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 2267a602306e4f248b73b62cf32bc6f3. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:29.448482 27262 tablet_service.cc:1022] Processing CreateTablet for tablet 76338445b4aa4c5abd99265bc7f96971 (table=ksck-test-table [id=15622109a20a45faa9a893f6023b1c6c]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:29.449460 27262 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 76338445b4aa4c5abd99265bc7f96971. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:29.459424 27397 tablet_bootstrap.cc:439] T 2267a602306e4f248b73b62cf32bc6f3 P 2d9779d348c74e069521dc32fa90746e: Bootstrap starting. I0918 20:39:29.469276 27397 tablet_bootstrap.cc:590] T 2267a602306e4f248b73b62cf32bc6f3 P 2d9779d348c74e069521dc32fa90746e: No blocks or log segments found. Creating new log. I0918 20:39:29.509989 27144 catalog_manager.cc:1153] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099: acquiring CA information for follower catalog manager: success I0918 20:39:29.515856 27145 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9: acquiring CA information for follower catalog manager: success I0918 20:39:29.524183 27399 tablet_bootstrap.cc:439] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9: Bootstrap starting. I0918 20:39:29.525017 27400 tablet_bootstrap.cc:439] T 76338445b4aa4c5abd99265bc7f96971 P 56bdddb1aabd482b86ac1fdc71e75369: Bootstrap starting. I0918 20:39:29.527746 27399 tablet_bootstrap.cc:590] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9: No blocks or log segments found. Creating new log. I0918 20:39:29.530838 27144 catalog_manager.cc:1181] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:29.537961 27397 tablet_bootstrap.cc:439] T 2267a602306e4f248b73b62cf32bc6f3 P 2d9779d348c74e069521dc32fa90746e: No bootstrap required, opened a new log I0918 20:39:29.540148 27397 ts_tablet_manager.cc:1085] T 2267a602306e4f248b73b62cf32bc6f3 P 2d9779d348c74e069521dc32fa90746e: Time spent bootstrapping tablet: real 0.081s user 0.010s sys 0.018s I0918 20:39:29.528961 27145 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:29.541203 27399 tablet_bootstrap.cc:439] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9: No bootstrap required, opened a new log I0918 20:39:29.539608 27400 tablet_bootstrap.cc:590] T 76338445b4aa4c5abd99265bc7f96971 P 56bdddb1aabd482b86ac1fdc71e75369: No blocks or log segments found. Creating new log. I0918 20:39:29.541646 27399 ts_tablet_manager.cc:1085] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.003s I0918 20:39:29.543799 27399 raft_consensus.cc:340] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:29.544497 27399 raft_consensus.cc:366] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:29.544711 27399 raft_consensus.cc:705] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df189ce5bd184e2dac9a57ccb83e6cf9, State: Initialized, Role: FOLLOWER I0918 20:39:29.543812 27397 raft_consensus.cc:340] T 2267a602306e4f248b73b62cf32bc6f3 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } I0918 20:39:29.547101 27397 raft_consensus.cc:366] T 2267a602306e4f248b73b62cf32bc6f3 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:29.547365 27397 raft_consensus.cc:705] T 2267a602306e4f248b73b62cf32bc6f3 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d9779d348c74e069521dc32fa90746e, State: Initialized, Role: FOLLOWER I0918 20:39:29.553987 27397 consensus_queue.cc:229] T 2267a602306e4f248b73b62cf32bc6f3 P 2d9779d348c74e069521dc32fa90746e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } I0918 20:39:29.545358 27399 consensus_queue.cc:229] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:29.566233 27400 tablet_bootstrap.cc:439] T 76338445b4aa4c5abd99265bc7f96971 P 56bdddb1aabd482b86ac1fdc71e75369: No bootstrap required, opened a new log I0918 20:39:29.574431 27397 ts_tablet_manager.cc:1113] T 2267a602306e4f248b73b62cf32bc6f3 P 2d9779d348c74e069521dc32fa90746e: Time spent starting tablet: real 0.034s user 0.006s sys 0.000s I0918 20:39:29.584431 27397 tablet_bootstrap.cc:439] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e: Bootstrap starting. I0918 20:39:29.583284 27400 ts_tablet_manager.cc:1085] T 76338445b4aa4c5abd99265bc7f96971 P 56bdddb1aabd482b86ac1fdc71e75369: Time spent bootstrapping tablet: real 0.058s user 0.011s sys 0.000s I0918 20:39:29.582810 27399 ts_tablet_manager.cc:1113] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9: Time spent starting tablet: real 0.041s user 0.004s sys 0.001s I0918 20:39:29.599341 27399 tablet_bootstrap.cc:439] T 2267a602306e4f248b73b62cf32bc6f3 P df189ce5bd184e2dac9a57ccb83e6cf9: Bootstrap starting. I0918 20:39:29.602403 27397 tablet_bootstrap.cc:590] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e: No blocks or log segments found. Creating new log. I0918 20:39:29.606566 27399 tablet_bootstrap.cc:590] T 2267a602306e4f248b73b62cf32bc6f3 P df189ce5bd184e2dac9a57ccb83e6cf9: No blocks or log segments found. Creating new log. I0918 20:39:29.605094 27400 raft_consensus.cc:340] T 76338445b4aa4c5abd99265bc7f96971 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:29.609529 27400 raft_consensus.cc:366] T 76338445b4aa4c5abd99265bc7f96971 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:29.616470 27400 raft_consensus.cc:705] T 76338445b4aa4c5abd99265bc7f96971 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56bdddb1aabd482b86ac1fdc71e75369, State: Initialized, Role: FOLLOWER I0918 20:39:29.617449 27400 consensus_queue.cc:229] T 76338445b4aa4c5abd99265bc7f96971 P 56bdddb1aabd482b86ac1fdc71e75369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:29.619354 27397 tablet_bootstrap.cc:439] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e: No bootstrap required, opened a new log I0918 20:39:29.620317 27397 ts_tablet_manager.cc:1085] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e: Time spent bootstrapping tablet: real 0.036s user 0.012s sys 0.002s I0918 20:39:29.622468 27400 ts_tablet_manager.cc:1113] T 76338445b4aa4c5abd99265bc7f96971 P 56bdddb1aabd482b86ac1fdc71e75369: Time spent starting tablet: real 0.025s user 0.005s sys 0.001s I0918 20:39:29.623817 27399 tablet_bootstrap.cc:439] T 2267a602306e4f248b73b62cf32bc6f3 P df189ce5bd184e2dac9a57ccb83e6cf9: No bootstrap required, opened a new log I0918 20:39:29.624405 27399 ts_tablet_manager.cc:1085] T 2267a602306e4f248b73b62cf32bc6f3 P df189ce5bd184e2dac9a57ccb83e6cf9: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.001s I0918 20:39:29.629748 27400 tablet_bootstrap.cc:439] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369: Bootstrap starting. I0918 20:39:29.634114 27400 tablet_bootstrap.cc:590] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369: No blocks or log segments found. Creating new log. I0918 20:39:29.635501 27397 raft_consensus.cc:340] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:29.636440 27397 raft_consensus.cc:366] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:29.636723 27397 raft_consensus.cc:705] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d9779d348c74e069521dc32fa90746e, State: Initialized, Role: FOLLOWER I0918 20:39:29.652495 27400 tablet_bootstrap.cc:439] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369: No bootstrap required, opened a new log I0918 20:39:29.653215 27400 ts_tablet_manager.cc:1085] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.008s I0918 20:39:29.655776 27400 raft_consensus.cc:340] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:29.656584 27400 raft_consensus.cc:366] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:29.656280 27399 raft_consensus.cc:340] T 2267a602306e4f248b73b62cf32bc6f3 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } I0918 20:39:29.656853 27400 raft_consensus.cc:705] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56bdddb1aabd482b86ac1fdc71e75369, State: Initialized, Role: FOLLOWER I0918 20:39:29.657783 27400 consensus_queue.cc:229] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:29.666268 27400 ts_tablet_manager.cc:1113] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369: Time spent starting tablet: real 0.013s user 0.005s sys 0.004s I0918 20:39:29.667181 27400 tablet_bootstrap.cc:439] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369: Bootstrap starting. I0918 20:39:29.670594 27400 tablet_bootstrap.cc:590] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369: No blocks or log segments found. Creating new log. I0918 20:39:29.675586 27400 tablet_bootstrap.cc:439] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369: No bootstrap required, opened a new log I0918 20:39:29.676190 27400 ts_tablet_manager.cc:1085] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s I0918 20:39:29.657524 27397 consensus_queue.cc:229] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:29.678342 27400 raft_consensus.cc:340] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } I0918 20:39:29.679085 27400 raft_consensus.cc:366] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:29.679334 27400 raft_consensus.cc:705] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56bdddb1aabd482b86ac1fdc71e75369, State: Initialized, Role: FOLLOWER I0918 20:39:29.680578 27397 ts_tablet_manager.cc:1113] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e: Time spent starting tablet: real 0.060s user 0.006s sys 0.000s I0918 20:39:29.680158 27400 consensus_queue.cc:229] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } I0918 20:39:29.681504 27397 tablet_bootstrap.cc:439] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e: Bootstrap starting. I0918 20:39:29.657213 27399 raft_consensus.cc:366] T 2267a602306e4f248b73b62cf32bc6f3 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:29.691206 27399 raft_consensus.cc:705] T 2267a602306e4f248b73b62cf32bc6f3 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df189ce5bd184e2dac9a57ccb83e6cf9, State: Initialized, Role: FOLLOWER I0918 20:39:29.691779 27400 ts_tablet_manager.cc:1113] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s I0918 20:39:29.692032 27399 consensus_queue.cc:229] T 2267a602306e4f248b73b62cf32bc6f3 P df189ce5bd184e2dac9a57ccb83e6cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } I0918 20:39:29.693871 27399 ts_tablet_manager.cc:1113] T 2267a602306e4f248b73b62cf32bc6f3 P df189ce5bd184e2dac9a57ccb83e6cf9: Time spent starting tablet: real 0.069s user 0.002s sys 0.004s I0918 20:39:29.694444 27399 tablet_bootstrap.cc:439] T 76338445b4aa4c5abd99265bc7f96971 P df189ce5bd184e2dac9a57ccb83e6cf9: Bootstrap starting. I0918 20:39:29.698226 27397 tablet_bootstrap.cc:590] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e: No blocks or log segments found. Creating new log. I0918 20:39:29.710661 27399 tablet_bootstrap.cc:590] T 76338445b4aa4c5abd99265bc7f96971 P df189ce5bd184e2dac9a57ccb83e6cf9: No blocks or log segments found. Creating new log. I0918 20:39:29.718757 27397 tablet_bootstrap.cc:439] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e: No bootstrap required, opened a new log I0918 20:39:29.720316 27399 tablet_bootstrap.cc:439] T 76338445b4aa4c5abd99265bc7f96971 P df189ce5bd184e2dac9a57ccb83e6cf9: No bootstrap required, opened a new log I0918 20:39:29.720894 27397 ts_tablet_manager.cc:1085] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.007s I0918 20:39:29.720953 27399 ts_tablet_manager.cc:1085] T 76338445b4aa4c5abd99265bc7f96971 P df189ce5bd184e2dac9a57ccb83e6cf9: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.001s I0918 20:39:29.723491 27397 raft_consensus.cc:340] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:29.723523 27403 raft_consensus.cc:472] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:29.724560 27403 raft_consensus.cc:494] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:29.726399 27403 leader_election.cc:296] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df189ce5bd184e2dac9a57ccb83e6cf9 (127.18.222.2:43213), 56bdddb1aabd482b86ac1fdc71e75369 (127.18.222.3:34229) I0918 20:39:29.726627 27399 raft_consensus.cc:340] T 76338445b4aa4c5abd99265bc7f96971 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:29.727418 27399 raft_consensus.cc:366] T 76338445b4aa4c5abd99265bc7f96971 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:29.727819 27399 raft_consensus.cc:705] T 76338445b4aa4c5abd99265bc7f96971 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df189ce5bd184e2dac9a57ccb83e6cf9, State: Initialized, Role: FOLLOWER I0918 20:39:29.729298 27397 raft_consensus.cc:366] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:29.729578 27397 raft_consensus.cc:705] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d9779d348c74e069521dc32fa90746e, State: Initialized, Role: FOLLOWER I0918 20:39:29.730347 27397 consensus_queue.cc:229] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:29.736287 27397 ts_tablet_manager.cc:1113] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s I0918 20:39:29.739060 27399 consensus_queue.cc:229] T 76338445b4aa4c5abd99265bc7f96971 P df189ce5bd184e2dac9a57ccb83e6cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:29.761255 27399 ts_tablet_manager.cc:1113] T 76338445b4aa4c5abd99265bc7f96971 P df189ce5bd184e2dac9a57ccb83e6cf9: Time spent starting tablet: real 0.040s user 0.005s sys 0.000s I0918 20:39:29.771025 27345 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "76338445b4aa4c5abd99265bc7f96971" candidate_uuid: "2d9779d348c74e069521dc32fa90746e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56bdddb1aabd482b86ac1fdc71e75369" is_pre_election: true I0918 20:39:29.772064 27345 raft_consensus.cc:2350] T 76338445b4aa4c5abd99265bc7f96971 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d9779d348c74e069521dc32fa90746e in term 0. I0918 20:39:29.772179 27274 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "76338445b4aa4c5abd99265bc7f96971" candidate_uuid: "2d9779d348c74e069521dc32fa90746e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" is_pre_election: true I0918 20:39:29.772840 27274 raft_consensus.cc:2350] T 76338445b4aa4c5abd99265bc7f96971 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d9779d348c74e069521dc32fa90746e in term 0. I0918 20:39:29.774716 27156 leader_election.cc:310] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d9779d348c74e069521dc32fa90746e, 56bdddb1aabd482b86ac1fdc71e75369; no voters: I0918 20:39:29.775974 27403 raft_consensus.cc:2644] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:29.776306 27403 raft_consensus.cc:472] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:29.776597 27403 raft_consensus.cc:2890] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:29.785295 27403 raft_consensus.cc:494] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:29.786975 27403 leader_election.cc:296] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e [CANDIDATE]: Term 1 election: Requested vote from peers df189ce5bd184e2dac9a57ccb83e6cf9 (127.18.222.2:43213), 56bdddb1aabd482b86ac1fdc71e75369 (127.18.222.3:34229) I0918 20:39:29.788273 27345 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "76338445b4aa4c5abd99265bc7f96971" candidate_uuid: "2d9779d348c74e069521dc32fa90746e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56bdddb1aabd482b86ac1fdc71e75369" I0918 20:39:29.788113 27274 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "76338445b4aa4c5abd99265bc7f96971" candidate_uuid: "2d9779d348c74e069521dc32fa90746e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" I0918 20:39:29.788849 27345 raft_consensus.cc:2890] T 76338445b4aa4c5abd99265bc7f96971 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:29.788949 27274 raft_consensus.cc:2890] T 76338445b4aa4c5abd99265bc7f96971 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:29.791841 27407 raft_consensus.cc:472] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:29.792408 27407 raft_consensus.cc:494] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } I0918 20:39:29.794350 27407 leader_election.cc:296] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df189ce5bd184e2dac9a57ccb83e6cf9 (127.18.222.2:43213), 2d9779d348c74e069521dc32fa90746e (127.18.222.1:41505) I0918 20:39:29.796839 27345 raft_consensus.cc:2350] T 76338445b4aa4c5abd99265bc7f96971 P 56bdddb1aabd482b86ac1fdc71e75369 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d9779d348c74e069521dc32fa90746e in term 1. I0918 20:39:29.799387 27274 raft_consensus.cc:2350] T 76338445b4aa4c5abd99265bc7f96971 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d9779d348c74e069521dc32fa90746e in term 1. I0918 20:39:29.794752 27407 raft_consensus.cc:472] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:29.802495 27407 raft_consensus.cc:494] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:29.812717 27407 leader_election.cc:296] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d9779d348c74e069521dc32fa90746e (127.18.222.1:41505), df189ce5bd184e2dac9a57ccb83e6cf9 (127.18.222.2:43213) I0918 20:39:29.817867 27156 leader_election.cc:310] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d9779d348c74e069521dc32fa90746e, 56bdddb1aabd482b86ac1fdc71e75369; no voters: I0918 20:39:29.810688 27403 raft_consensus.cc:472] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:29.826571 27403 raft_consensus.cc:494] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:29.835198 27403 leader_election.cc:296] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df189ce5bd184e2dac9a57ccb83e6cf9 (127.18.222.2:43213), 56bdddb1aabd482b86ac1fdc71e75369 (127.18.222.3:34229) I0918 20:39:29.835724 27403 raft_consensus.cc:2644] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:29.836232 27403 raft_consensus.cc:667] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e [term 1 LEADER]: Becoming Leader. State: Replica: 2d9779d348c74e069521dc32fa90746e, State: Running, Role: LEADER I0918 20:39:29.836884 27403 consensus_queue.cc:206] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:29.840471 27404 raft_consensus.cc:472] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:29.841114 27404 raft_consensus.cc:494] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:29.844316 27404 leader_election.cc:296] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d9779d348c74e069521dc32fa90746e (127.18.222.1:41505), 56bdddb1aabd482b86ac1fdc71e75369 (127.18.222.3:34229) I0918 20:39:29.855226 27417 raft_consensus.cc:472] T 2267a602306e4f248b73b62cf32bc6f3 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:29.865165 27345 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3e240c7a9d654cac8b05288625596c49" candidate_uuid: "2d9779d348c74e069521dc32fa90746e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56bdddb1aabd482b86ac1fdc71e75369" is_pre_election: true I0918 20:39:29.866008 27345 raft_consensus.cc:2350] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d9779d348c74e069521dc32fa90746e in term 0. I0918 20:39:29.869947 27274 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3e240c7a9d654cac8b05288625596c49" candidate_uuid: "2d9779d348c74e069521dc32fa90746e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" is_pre_election: true I0918 20:39:29.870604 27274 raft_consensus.cc:2350] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d9779d348c74e069521dc32fa90746e in term 0. I0918 20:39:29.862962 27417 raft_consensus.cc:494] T 2267a602306e4f248b73b62cf32bc6f3 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } I0918 20:39:29.877228 27417 leader_election.cc:296] T 2267a602306e4f248b73b62cf32bc6f3 P 2d9779d348c74e069521dc32fa90746e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56bdddb1aabd482b86ac1fdc71e75369 (127.18.222.3:34229), df189ce5bd184e2dac9a57ccb83e6cf9 (127.18.222.2:43213) I0918 20:39:29.875526 27156 leader_election.cc:310] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d9779d348c74e069521dc32fa90746e, 56bdddb1aabd482b86ac1fdc71e75369; no voters: I0918 20:39:29.881844 27203 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "2267a602306e4f248b73b62cf32bc6f3" candidate_uuid: "56bdddb1aabd482b86ac1fdc71e75369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d9779d348c74e069521dc32fa90746e" is_pre_election: true I0918 20:39:29.882726 27203 raft_consensus.cc:2350] T 2267a602306e4f248b73b62cf32bc6f3 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56bdddb1aabd482b86ac1fdc71e75369 in term 0. I0918 20:39:29.884394 27306 leader_election.cc:310] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d9779d348c74e069521dc32fa90746e, 56bdddb1aabd482b86ac1fdc71e75369; no voters: I0918 20:39:29.884724 27203 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3e240c7a9d654cac8b05288625596c49" candidate_uuid: "56bdddb1aabd482b86ac1fdc71e75369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d9779d348c74e069521dc32fa90746e" is_pre_election: true I0918 20:39:29.885159 27417 raft_consensus.cc:2644] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:29.887552 27203 raft_consensus.cc:2350] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56bdddb1aabd482b86ac1fdc71e75369 in term 0. I0918 20:39:29.888368 27414 raft_consensus.cc:2644] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:29.888767 27414 raft_consensus.cc:472] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:29.889097 27414 raft_consensus.cc:2890] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:29.892244 27417 raft_consensus.cc:472] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:29.893048 27417 raft_consensus.cc:2890] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:29.895817 27414 raft_consensus.cc:494] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } I0918 20:39:29.898201 27414 leader_election.cc:296] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [CANDIDATE]: Term 1 election: Requested vote from peers df189ce5bd184e2dac9a57ccb83e6cf9 (127.18.222.2:43213), 2d9779d348c74e069521dc32fa90746e (127.18.222.1:41505) I0918 20:39:29.899266 27306 leader_election.cc:310] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d9779d348c74e069521dc32fa90746e, 56bdddb1aabd482b86ac1fdc71e75369; no voters: I0918 20:39:29.901199 27102 catalog_manager.cc:4131] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e reported cstate change: term changed from 0 to 1, leader changed from to 2d9779d348c74e069521dc32fa90746e (127.18.222.1). New cstate: current_term: 1 leader_uuid: "2d9779d348c74e069521dc32fa90746e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } health_report { overall_health: UNKNOWN } } } I0918 20:39:29.902534 27203 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "2267a602306e4f248b73b62cf32bc6f3" candidate_uuid: "56bdddb1aabd482b86ac1fdc71e75369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d9779d348c74e069521dc32fa90746e" I0918 20:39:29.903218 27203 raft_consensus.cc:2890] T 2267a602306e4f248b73b62cf32bc6f3 P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:29.903760 27274 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "2267a602306e4f248b73b62cf32bc6f3" candidate_uuid: "2d9779d348c74e069521dc32fa90746e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" is_pre_election: true I0918 20:39:29.905215 27274 raft_consensus.cc:2350] T 2267a602306e4f248b73b62cf32bc6f3 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d9779d348c74e069521dc32fa90746e in term 0. I0918 20:39:29.907094 27345 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "2267a602306e4f248b73b62cf32bc6f3" candidate_uuid: "2d9779d348c74e069521dc32fa90746e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56bdddb1aabd482b86ac1fdc71e75369" is_pre_election: true I0918 20:39:29.910163 27345 raft_consensus.cc:2280] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2d9779d348c74e069521dc32fa90746e in current term 1: Already voted for candidate 56bdddb1aabd482b86ac1fdc71e75369 in this term. I0918 20:39:29.916054 27156 leader_election.cc:310] T 2267a602306e4f248b73b62cf32bc6f3 P 2d9779d348c74e069521dc32fa90746e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d9779d348c74e069521dc32fa90746e, df189ce5bd184e2dac9a57ccb83e6cf9; no voters: I0918 20:39:29.918745 27414 raft_consensus.cc:2644] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:29.925398 27414 raft_consensus.cc:472] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:29.920277 27272 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "2267a602306e4f248b73b62cf32bc6f3" candidate_uuid: "56bdddb1aabd482b86ac1fdc71e75369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" I0918 20:39:29.925892 27272 raft_consensus.cc:2890] T 2267a602306e4f248b73b62cf32bc6f3 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:29.920455 27273 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3e240c7a9d654cac8b05288625596c49" candidate_uuid: "56bdddb1aabd482b86ac1fdc71e75369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" is_pre_election: true I0918 20:39:29.928305 27273 raft_consensus.cc:2350] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56bdddb1aabd482b86ac1fdc71e75369 in term 0. I0918 20:39:29.921111 27274 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "2267a602306e4f248b73b62cf32bc6f3" candidate_uuid: "56bdddb1aabd482b86ac1fdc71e75369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" is_pre_election: true I0918 20:39:29.919282 27203 raft_consensus.cc:2350] T 2267a602306e4f248b73b62cf32bc6f3 P 2d9779d348c74e069521dc32fa90746e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56bdddb1aabd482b86ac1fdc71e75369 in term 1. I0918 20:39:29.930204 27414 raft_consensus.cc:2890] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:29.932703 27419 raft_consensus.cc:2607] T 2267a602306e4f248b73b62cf32bc6f3 P 2d9779d348c74e069521dc32fa90746e [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won I0918 20:39:29.933562 27306 leader_election.cc:310] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d9779d348c74e069521dc32fa90746e, 56bdddb1aabd482b86ac1fdc71e75369; no voters: I0918 20:39:29.936813 27407 raft_consensus.cc:2644] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:29.954444 27272 raft_consensus.cc:2350] T 2267a602306e4f248b73b62cf32bc6f3 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56bdddb1aabd482b86ac1fdc71e75369 in term 1. I0918 20:39:29.955070 27274 raft_consensus.cc:2325] T 2267a602306e4f248b73b62cf32bc6f3 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 56bdddb1aabd482b86ac1fdc71e75369 for term 1 because replica is already servicing an update from a current leader or another vote. I0918 20:39:29.954732 27417 raft_consensus.cc:494] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:29.960443 27345 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3e240c7a9d654cac8b05288625596c49" candidate_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56bdddb1aabd482b86ac1fdc71e75369" is_pre_election: true I0918 20:39:29.963403 27414 raft_consensus.cc:494] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:29.966075 27345 raft_consensus.cc:2280] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df189ce5bd184e2dac9a57ccb83e6cf9 in current term 1: Already voted for candidate 56bdddb1aabd482b86ac1fdc71e75369 in this term. I0918 20:39:29.969794 27407 raft_consensus.cc:667] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [term 1 LEADER]: Becoming Leader. State: Replica: 56bdddb1aabd482b86ac1fdc71e75369, State: Running, Role: LEADER I0918 20:39:29.972496 27407 consensus_queue.cc:206] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } I0918 20:39:29.978441 27274 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3e240c7a9d654cac8b05288625596c49" candidate_uuid: "2d9779d348c74e069521dc32fa90746e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" I0918 20:39:29.978966 27417 leader_election.cc:296] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [CANDIDATE]: Term 1 election: Requested vote from peers df189ce5bd184e2dac9a57ccb83e6cf9 (127.18.222.2:43213), 56bdddb1aabd482b86ac1fdc71e75369 (127.18.222.3:34229) I0918 20:39:29.979240 27274 raft_consensus.cc:2890] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:29.979179 27345 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3e240c7a9d654cac8b05288625596c49" candidate_uuid: "2d9779d348c74e069521dc32fa90746e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56bdddb1aabd482b86ac1fdc71e75369" I0918 20:39:29.980269 27345 raft_consensus.cc:2280] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2d9779d348c74e069521dc32fa90746e in current term 1: Already voted for candidate 56bdddb1aabd482b86ac1fdc71e75369 in this term. I0918 20:39:29.986183 27203 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3e240c7a9d654cac8b05288625596c49" candidate_uuid: "56bdddb1aabd482b86ac1fdc71e75369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d9779d348c74e069521dc32fa90746e" I0918 20:39:29.986431 27274 raft_consensus.cc:2350] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d9779d348c74e069521dc32fa90746e in term 1. I0918 20:39:29.987018 27203 raft_consensus.cc:2280] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 56bdddb1aabd482b86ac1fdc71e75369 in current term 1: Already voted for candidate 2d9779d348c74e069521dc32fa90746e in this term. I0918 20:39:29.987911 27156 leader_election.cc:310] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d9779d348c74e069521dc32fa90746e, df189ce5bd184e2dac9a57ccb83e6cf9; no voters: 56bdddb1aabd482b86ac1fdc71e75369 I0918 20:39:29.986958 27272 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3e240c7a9d654cac8b05288625596c49" candidate_uuid: "56bdddb1aabd482b86ac1fdc71e75369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" I0918 20:39:29.988847 27417 raft_consensus.cc:2644] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:29.988955 27272 raft_consensus.cc:2280] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 56bdddb1aabd482b86ac1fdc71e75369 in current term 1: Already voted for candidate 2d9779d348c74e069521dc32fa90746e in this term. I0918 20:39:29.989656 27417 raft_consensus.cc:667] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [term 1 LEADER]: Becoming Leader. State: Replica: 2d9779d348c74e069521dc32fa90746e, State: Running, Role: LEADER I0918 20:39:29.990206 27308 leader_election.cc:310] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56bdddb1aabd482b86ac1fdc71e75369; no voters: 2d9779d348c74e069521dc32fa90746e, df189ce5bd184e2dac9a57ccb83e6cf9 I0918 20:39:29.990470 27417 consensus_queue.cc:206] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:29.991051 27424 raft_consensus.cc:2592] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority I0918 20:39:29.991850 27414 leader_election.cc:296] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369 [CANDIDATE]: Term 1 election: Requested vote from peers 2d9779d348c74e069521dc32fa90746e (127.18.222.1:41505), df189ce5bd184e2dac9a57ccb83e6cf9 (127.18.222.2:43213) I0918 20:39:29.994909 27203 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3e240c7a9d654cac8b05288625596c49" candidate_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d9779d348c74e069521dc32fa90746e" is_pre_election: true I0918 20:39:29.995987 27203 raft_consensus.cc:2311] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate df189ce5bd184e2dac9a57ccb83e6cf9 for term 1 because replica is either leader or believes a valid leader to be alive. I0918 20:39:29.997913 27235 leader_election.cc:310] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df189ce5bd184e2dac9a57ccb83e6cf9; no voters: 2d9779d348c74e069521dc32fa90746e, 56bdddb1aabd482b86ac1fdc71e75369 I0918 20:39:29.999456 27404 raft_consensus.cc:2592] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0918 20:39:30.048161 27103 catalog_manager.cc:4131] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 reported cstate change: term changed from 0 to 1, leader changed from to 56bdddb1aabd482b86ac1fdc71e75369 (127.18.222.3). New cstate: current_term: 1 leader_uuid: "56bdddb1aabd482b86ac1fdc71e75369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } health_report { overall_health: UNKNOWN } } } I0918 20:39:30.080466 27102 catalog_manager.cc:4131] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e reported cstate change: term changed from 0 to 1, leader changed from to 2d9779d348c74e069521dc32fa90746e (127.18.222.1). New cstate: current_term: 1 leader_uuid: "2d9779d348c74e069521dc32fa90746e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } health_report { overall_health: UNKNOWN } } } I0918 20:39:30.239022 27101 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39322: name: "other-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "value" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\006\001\000\000\000\000\007\001d\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:39:30.240851 27101 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table other-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:39:30.267529 27417 consensus_queue.cc:984] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:30.289960 27262 tablet_service.cc:1022] Processing CreateTablet for tablet ca12ca1278624f6cb468ce657492ed5c (table=other-table [id=1e659a8616b040e3aca13aae148442b2]), partition=RANGE (key) PARTITION 0 <= VALUES < 100 I0918 20:39:30.290863 27333 tablet_service.cc:1022] Processing CreateTablet for tablet ca12ca1278624f6cb468ce657492ed5c (table=other-table [id=1e659a8616b040e3aca13aae148442b2]), partition=RANGE (key) PARTITION 0 <= VALUES < 100 I0918 20:39:30.291532 27189 tablet_service.cc:1022] Processing CreateTablet for tablet ca12ca1278624f6cb468ce657492ed5c (table=other-table [id=1e659a8616b040e3aca13aae148442b2]), partition=RANGE (key) PARTITION 0 <= VALUES < 100 I0918 20:39:30.293462 27189 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet ca12ca1278624f6cb468ce657492ed5c. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:30.291224 27262 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet ca12ca1278624f6cb468ce657492ed5c. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:30.305708 27333 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet ca12ca1278624f6cb468ce657492ed5c. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:30.364595 27440 tablet_bootstrap.cc:439] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9: Bootstrap starting. I0918 20:39:30.375171 27417 consensus_queue.cc:984] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:39:30.408043 27440 tablet_bootstrap.cc:590] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9: No blocks or log segments found. Creating new log. I0918 20:39:30.453536 27443 tablet_bootstrap.cc:439] T ca12ca1278624f6cb468ce657492ed5c P 56bdddb1aabd482b86ac1fdc71e75369: Bootstrap starting. W0918 20:39:30.446271 27156 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.222.2:43213, user_credentials={real_user=slave}} blocked reactor thread for 49685.5us I0918 20:39:30.458171 27407 consensus_queue.cc:984] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:30.475448 27445 consensus_queue.cc:984] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s I0918 20:39:30.479990 27440 tablet_bootstrap.cc:439] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9: No bootstrap required, opened a new log I0918 20:39:30.480828 27440 ts_tablet_manager.cc:1085] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9: Time spent bootstrapping tablet: real 0.116s user 0.017s sys 0.029s I0918 20:39:30.482782 27445 consensus_queue.cc:984] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:39:30.485824 27407 consensus_queue.cc:984] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:30.513655 27443 tablet_bootstrap.cc:590] T ca12ca1278624f6cb468ce657492ed5c P 56bdddb1aabd482b86ac1fdc71e75369: No blocks or log segments found. Creating new log. I0918 20:39:30.521479 27440 raft_consensus.cc:340] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:30.523433 27440 raft_consensus.cc:366] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:30.523751 27440 raft_consensus.cc:705] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df189ce5bd184e2dac9a57ccb83e6cf9, State: Initialized, Role: FOLLOWER I0918 20:39:30.524606 27440 consensus_queue.cc:229] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:30.526844 27440 ts_tablet_manager.cc:1113] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9: Time spent starting tablet: real 0.046s user 0.006s sys 0.000s I0918 20:39:30.538851 27448 tablet_bootstrap.cc:439] T ca12ca1278624f6cb468ce657492ed5c P 2d9779d348c74e069521dc32fa90746e: Bootstrap starting. I0918 20:39:30.543496 27448 tablet_bootstrap.cc:590] T ca12ca1278624f6cb468ce657492ed5c P 2d9779d348c74e069521dc32fa90746e: No blocks or log segments found. Creating new log. I0918 20:39:30.601167 27443 tablet_bootstrap.cc:439] T ca12ca1278624f6cb468ce657492ed5c P 56bdddb1aabd482b86ac1fdc71e75369: No bootstrap required, opened a new log I0918 20:39:30.602046 27443 ts_tablet_manager.cc:1085] T ca12ca1278624f6cb468ce657492ed5c P 56bdddb1aabd482b86ac1fdc71e75369: Time spent bootstrapping tablet: real 0.152s user 0.016s sys 0.014s I0918 20:39:30.604400 27443 raft_consensus.cc:340] T ca12ca1278624f6cb468ce657492ed5c P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:30.605204 27443 raft_consensus.cc:366] T ca12ca1278624f6cb468ce657492ed5c P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:30.605504 27443 raft_consensus.cc:705] T ca12ca1278624f6cb468ce657492ed5c P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56bdddb1aabd482b86ac1fdc71e75369, State: Initialized, Role: FOLLOWER I0918 20:39:30.606251 27443 consensus_queue.cc:229] T ca12ca1278624f6cb468ce657492ed5c P 56bdddb1aabd482b86ac1fdc71e75369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:30.612146 27443 ts_tablet_manager.cc:1113] T ca12ca1278624f6cb468ce657492ed5c P 56bdddb1aabd482b86ac1fdc71e75369: Time spent starting tablet: real 0.010s user 0.003s sys 0.003s I0918 20:39:30.630581 27448 tablet_bootstrap.cc:439] T ca12ca1278624f6cb468ce657492ed5c P 2d9779d348c74e069521dc32fa90746e: No bootstrap required, opened a new log I0918 20:39:30.631290 27448 ts_tablet_manager.cc:1085] T ca12ca1278624f6cb468ce657492ed5c P 2d9779d348c74e069521dc32fa90746e: Time spent bootstrapping tablet: real 0.093s user 0.018s sys 0.005s I0918 20:39:30.633664 27448 raft_consensus.cc:340] T ca12ca1278624f6cb468ce657492ed5c P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:30.634517 27448 raft_consensus.cc:366] T ca12ca1278624f6cb468ce657492ed5c P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:30.634807 27448 raft_consensus.cc:705] T ca12ca1278624f6cb468ce657492ed5c P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d9779d348c74e069521dc32fa90746e, State: Initialized, Role: FOLLOWER I0918 20:39:30.639230 27448 consensus_queue.cc:229] T ca12ca1278624f6cb468ce657492ed5c P 2d9779d348c74e069521dc32fa90746e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:30.656419 27448 ts_tablet_manager.cc:1113] T ca12ca1278624f6cb468ce657492ed5c P 2d9779d348c74e069521dc32fa90746e: Time spent starting tablet: real 0.025s user 0.008s sys 0.000s I0918 20:39:30.677886 27404 raft_consensus.cc:472] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:30.678426 27404 raft_consensus.cc:494] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:30.680279 27404 leader_election.cc:296] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d9779d348c74e069521dc32fa90746e (127.18.222.1:41505), 56bdddb1aabd482b86ac1fdc71e75369 (127.18.222.3:34229) I0918 20:39:30.681085 27203 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ca12ca1278624f6cb468ce657492ed5c" candidate_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d9779d348c74e069521dc32fa90746e" is_pre_election: true I0918 20:39:30.681728 27344 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ca12ca1278624f6cb468ce657492ed5c" candidate_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56bdddb1aabd482b86ac1fdc71e75369" is_pre_election: true I0918 20:39:30.681871 27203 raft_consensus.cc:2350] T ca12ca1278624f6cb468ce657492ed5c P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df189ce5bd184e2dac9a57ccb83e6cf9 in term 0. I0918 20:39:30.682510 27344 raft_consensus.cc:2350] T ca12ca1278624f6cb468ce657492ed5c P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df189ce5bd184e2dac9a57ccb83e6cf9 in term 0. I0918 20:39:30.683382 27235 leader_election.cc:310] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d9779d348c74e069521dc32fa90746e, df189ce5bd184e2dac9a57ccb83e6cf9; no voters: I0918 20:39:30.684950 27404 raft_consensus.cc:2644] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:30.685257 27404 raft_consensus.cc:472] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:30.685497 27404 raft_consensus.cc:2890] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:30.692189 27404 raft_consensus.cc:494] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:30.694623 27203 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ca12ca1278624f6cb468ce657492ed5c" candidate_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d9779d348c74e069521dc32fa90746e" I0918 20:39:30.695099 27344 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ca12ca1278624f6cb468ce657492ed5c" candidate_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56bdddb1aabd482b86ac1fdc71e75369" I0918 20:39:30.695271 27203 raft_consensus.cc:2890] T ca12ca1278624f6cb468ce657492ed5c P 2d9779d348c74e069521dc32fa90746e [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:30.695843 27344 raft_consensus.cc:2890] T ca12ca1278624f6cb468ce657492ed5c P 56bdddb1aabd482b86ac1fdc71e75369 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:30.701761 27344 raft_consensus.cc:2350] T ca12ca1278624f6cb468ce657492ed5c P 56bdddb1aabd482b86ac1fdc71e75369 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df189ce5bd184e2dac9a57ccb83e6cf9 in term 1. I0918 20:39:30.702946 27237 leader_election.cc:310] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56bdddb1aabd482b86ac1fdc71e75369, df189ce5bd184e2dac9a57ccb83e6cf9; no voters: I0918 20:39:30.703311 27203 raft_consensus.cc:2350] T ca12ca1278624f6cb468ce657492ed5c P 2d9779d348c74e069521dc32fa90746e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df189ce5bd184e2dac9a57ccb83e6cf9 in term 1. I0918 20:39:30.705813 27404 leader_election.cc:296] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9 [CANDIDATE]: Term 1 election: Requested vote from peers 2d9779d348c74e069521dc32fa90746e (127.18.222.1:41505), 56bdddb1aabd482b86ac1fdc71e75369 (127.18.222.3:34229) I0918 20:39:30.713827 27404 raft_consensus.cc:2644] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:30.714294 27404 raft_consensus.cc:667] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9 [term 1 LEADER]: Becoming Leader. State: Replica: df189ce5bd184e2dac9a57ccb83e6cf9, State: Running, Role: LEADER I0918 20:39:30.714934 27404 consensus_queue.cc:206] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } } I0918 20:39:30.723495 27100 catalog_manager.cc:4131] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9 reported cstate change: term changed from 0 to 1, leader changed from to df189ce5bd184e2dac9a57ccb83e6cf9 (127.18.222.2). New cstate: current_term: 1 leader_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9779d348c74e069521dc32fa90746e" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df189ce5bd184e2dac9a57ccb83e6cf9" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56bdddb1aabd482b86ac1fdc71e75369" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 34229 } health_report { overall_health: UNKNOWN } } } I0918 20:39:30.779870 27100 catalog_manager.cc:2376] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39322: table { table_name: "other-table" } alter_schema_steps { type: DROP_RANGE_PARTITION drop_range_partition { range_bounds { rows: "\006\001\000\000\000\000\007\001d\000\000\000" } } } schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "value" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } modify_external_catalogs: true I0918 20:39:30.798408 27100 catalog_manager.cc:4381] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25: Sending DeleteTablet for 3 replicas of tablet ca12ca1278624f6cb468ce657492ed5c I0918 20:39:30.798960 27100 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ca12ca1278624f6cb468ce657492ed5c on 2d9779d348c74e069521dc32fa90746e (127.18.222.1:41505) (Partition dropped at 2019-09-18 20:39:30 UTC) I0918 20:39:30.800671 27100 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ca12ca1278624f6cb468ce657492ed5c on df189ce5bd184e2dac9a57ccb83e6cf9 (127.18.222.2:43213) (Partition dropped at 2019-09-18 20:39:30 UTC) I0918 20:39:30.801868 27189 tablet_service.cc:1066] Processing DeleteTablet for tablet ca12ca1278624f6cb468ce657492ed5c with delete_type TABLET_DATA_DELETED (Partition dropped at 2019-09-18 20:39:30 UTC) from {username='slave'} at 127.0.0.1:32966 I0918 20:39:30.802116 27262 tablet_service.cc:1066] Processing DeleteTablet for tablet ca12ca1278624f6cb468ce657492ed5c with delete_type TABLET_DATA_DELETED (Partition dropped at 2019-09-18 20:39:30 UTC) from {username='slave'} at 127.0.0.1:47656 I0918 20:39:30.802862 27100 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ca12ca1278624f6cb468ce657492ed5c on 56bdddb1aabd482b86ac1fdc71e75369 (127.18.222.3:34229) (Partition dropped at 2019-09-18 20:39:30 UTC) I0918 20:39:30.823933 27333 tablet_service.cc:1066] Processing DeleteTablet for tablet ca12ca1278624f6cb468ce657492ed5c with delete_type TABLET_DATA_DELETED (Partition dropped at 2019-09-18 20:39:30 UTC) from {username='slave'} at 127.0.0.1:47068 I0918 20:39:30.838809 27461 tablet_replica.cc:273] T ca12ca1278624f6cb468ce657492ed5c P 56bdddb1aabd482b86ac1fdc71e75369: stopping tablet replica I0918 20:39:30.839548 27461 raft_consensus.cc:2147] T ca12ca1278624f6cb468ce657492ed5c P 56bdddb1aabd482b86ac1fdc71e75369 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:30.840135 27461 raft_consensus.cc:2174] T ca12ca1278624f6cb468ce657492ed5c P 56bdddb1aabd482b86ac1fdc71e75369 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:30.842813 27459 tablet_replica.cc:273] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9: stopping tablet replica I0918 20:39:30.843575 27459 raft_consensus.cc:2147] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:30.844372 27459 pending_rounds.cc:64] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9: Trying to abort 1 pending transactions. I0918 20:39:30.844609 27459 pending_rounds.cc:70] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6425965243254665216 op_type: NO_OP noop_request { } I0918 20:39:30.845010 27459 raft_consensus.cc:2727] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted I0918 20:39:30.845293 27459 raft_consensus.cc:2174] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:30.845877 27460 tablet_replica.cc:273] T ca12ca1278624f6cb468ce657492ed5c P 2d9779d348c74e069521dc32fa90746e: stopping tablet replica I0918 20:39:30.846627 27460 raft_consensus.cc:2147] T ca12ca1278624f6cb468ce657492ed5c P 2d9779d348c74e069521dc32fa90746e [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:30.847035 27460 raft_consensus.cc:2174] T ca12ca1278624f6cb468ce657492ed5c P 2d9779d348c74e069521dc32fa90746e [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:30.865749 27459 ts_tablet_manager.cc:1440] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9: Deleting tablet data with delete state TABLET_DATA_DELETED I0918 20:39:30.875654 27460 ts_tablet_manager.cc:1440] T ca12ca1278624f6cb468ce657492ed5c P 2d9779d348c74e069521dc32fa90746e: Deleting tablet data with delete state TABLET_DATA_DELETED I0918 20:39:30.878295 27461 ts_tablet_manager.cc:1440] T ca12ca1278624f6cb468ce657492ed5c P 56bdddb1aabd482b86ac1fdc71e75369: Deleting tablet data with delete state TABLET_DATA_DELETED I0918 20:39:30.883574 27459 ts_tablet_manager.cc:1453] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 I0918 20:39:30.884151 27459 log.cc:990] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9: Deleting WAL directory at /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/ts-1-root/wals/ca12ca1278624f6cb468ce657492ed5c I0918 20:39:30.884984 27459 ts_tablet_manager.cc:1474] T ca12ca1278624f6cb468ce657492ed5c P df189ce5bd184e2dac9a57ccb83e6cf9: Deleting consensus metadata I0918 20:39:30.893991 27460 ts_tablet_manager.cc:1453] T ca12ca1278624f6cb468ce657492ed5c P 2d9779d348c74e069521dc32fa90746e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0 I0918 20:39:30.894479 27460 log.cc:990] T ca12ca1278624f6cb468ce657492ed5c P 2d9779d348c74e069521dc32fa90746e: Deleting WAL directory at /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/ts-0-root/wals/ca12ca1278624f6cb468ce657492ed5c I0918 20:39:30.895139 27460 ts_tablet_manager.cc:1474] T ca12ca1278624f6cb468ce657492ed5c P 2d9779d348c74e069521dc32fa90746e: Deleting consensus metadata I0918 20:39:30.960242 27086 catalog_manager.cc:3501] TS df189ce5bd184e2dac9a57ccb83e6cf9 (127.18.222.2:43213): tablet ca12ca1278624f6cb468ce657492ed5c (table other-table [id=1e659a8616b040e3aca13aae148442b2]) successfully deleted I0918 20:39:31.041057 27084 catalog_manager.cc:3501] TS 2d9779d348c74e069521dc32fa90746e (127.18.222.1:41505): tablet ca12ca1278624f6cb468ce657492ed5c (table other-table [id=1e659a8616b040e3aca13aae148442b2]) successfully deleted I0918 20:39:31.056996 27461 ts_tablet_manager.cc:1453] T ca12ca1278624f6cb468ce657492ed5c P 56bdddb1aabd482b86ac1fdc71e75369: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0 I0918 20:39:31.058310 27461 log.cc:990] T ca12ca1278624f6cb468ce657492ed5c P 56bdddb1aabd482b86ac1fdc71e75369: Deleting WAL directory at /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1568839089958813-19320/minicluster-data/ts-2-root/wals/ca12ca1278624f6cb468ce657492ed5c I0918 20:39:31.059494 27461 ts_tablet_manager.cc:1474] T ca12ca1278624f6cb468ce657492ed5c P 56bdddb1aabd482b86ac1fdc71e75369: Deleting consensus metadata W0918 20:39:31.069355 27156 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.222.3:34229, user_credentials={real_user=slave}} blocked reactor thread for 77571.8us I0918 20:39:31.112017 27086 catalog_manager.cc:3501] TS 56bdddb1aabd482b86ac1fdc71e75369 (127.18.222.3:34229): tablet ca12ca1278624f6cb468ce657492ed5c (table other-table [id=1e659a8616b040e3aca13aae148442b2]) successfully deleted I0918 20:39:31.611160 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:31.683497 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:31.683889 19320 tablet_replica.cc:273] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e: stopping tablet replica I0918 20:39:31.684320 19320 raft_consensus.cc:2147] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:31.684993 19320 raft_consensus.cc:2174] T 76338445b4aa4c5abd99265bc7f96971 P 2d9779d348c74e069521dc32fa90746e [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:31.687844 19320 tablet_replica.cc:273] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e: stopping tablet replica I0918 20:39:31.688210 19320 raft_consensus.cc:2147] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:31.688843 19320 raft_consensus.cc:2174] T 3e240c7a9d654cac8b05288625596c49 P 2d9779d348c74e069521dc32fa90746e [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:31.691663 19320 tablet_replica.cc:273] T 2267a602306e4f248b73b62cf32bc6f3 P 2d9779d348c74e069521dc32fa90746e: stopping tablet replica I0918 20:39:31.692203 19320 raft_consensus.cc:2147] T 2267a602306e4f248b73b62cf32bc6f3 P 2d9779d348c74e069521dc32fa90746e [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:31.692597 19320 raft_consensus.cc:2174] T 2267a602306e4f248b73b62cf32bc6f3 P 2d9779d348c74e069521dc32fa90746e [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:31.714372 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:31.714581 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:31.714907 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:31.734800 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:31.790148 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:31.790525 19320 tablet_replica.cc:273] T 76338445b4aa4c5abd99265bc7f96971 P df189ce5bd184e2dac9a57ccb83e6cf9: stopping tablet replica I0918 20:39:31.790932 19320 raft_consensus.cc:2147] T 76338445b4aa4c5abd99265bc7f96971 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:31.791224 19320 raft_consensus.cc:2174] T 76338445b4aa4c5abd99265bc7f96971 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:31.793486 19320 tablet_replica.cc:273] T 2267a602306e4f248b73b62cf32bc6f3 P df189ce5bd184e2dac9a57ccb83e6cf9: stopping tablet replica I0918 20:39:31.793848 19320 raft_consensus.cc:2147] T 2267a602306e4f248b73b62cf32bc6f3 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:31.794119 19320 raft_consensus.cc:2174] T 2267a602306e4f248b73b62cf32bc6f3 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:31.796245 19320 tablet_replica.cc:273] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9: stopping tablet replica I0918 20:39:31.796569 19320 raft_consensus.cc:2147] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:31.796856 19320 raft_consensus.cc:2174] T 3e240c7a9d654cac8b05288625596c49 P df189ce5bd184e2dac9a57ccb83e6cf9 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:31.821133 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:31.821329 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:31.821681 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:31.841456 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:31.885550 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:31.885895 19320 tablet_replica.cc:273] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369: stopping tablet replica I0918 20:39:31.886289 19320 raft_consensus.cc:2147] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:31.886981 19320 raft_consensus.cc:2174] T 2267a602306e4f248b73b62cf32bc6f3 P 56bdddb1aabd482b86ac1fdc71e75369 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:31.889194 19320 tablet_replica.cc:273] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369: stopping tablet replica I0918 20:39:31.889560 19320 raft_consensus.cc:2147] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:31.889833 19320 raft_consensus.cc:2174] T 3e240c7a9d654cac8b05288625596c49 P 56bdddb1aabd482b86ac1fdc71e75369 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:31.891963 19320 tablet_replica.cc:273] T 76338445b4aa4c5abd99265bc7f96971 P 56bdddb1aabd482b86ac1fdc71e75369: stopping tablet replica I0918 20:39:31.892338 19320 raft_consensus.cc:2147] T 76338445b4aa4c5abd99265bc7f96971 P 56bdddb1aabd482b86ac1fdc71e75369 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:31.892627 19320 raft_consensus.cc:2174] T 76338445b4aa4c5abd99265bc7f96971 P 56bdddb1aabd482b86ac1fdc71e75369 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:31.914629 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:31.914858 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:31.915235 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:31.930869 19320 master.cc:247] Master@127.18.222.62:44123 shutting down... I0918 20:39:31.950433 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:31.950845 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:31.951093 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 065d5720fcff45859a00631da55c7ef9: stopping tablet replica I0918 20:39:31.970044 19320 master.cc:258] Master@127.18.222.62:44123 shutdown complete. I0918 20:39:31.980008 19320 master.cc:247] Master@127.18.222.61:40549 shutting down... I0918 20:39:32.006440 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:32.006857 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:32.007115 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P a35f07dedf474aab830daea0ff904099: stopping tablet replica I0918 20:39:32.025712 19320 master.cc:258] Master@127.18.222.61:40549 shutdown complete. I0918 20:39:32.037431 19320 master.cc:247] Master@127.18.222.60:32967 shutting down... I0918 20:39:32.059043 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:32.059939 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:32.060210 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 3514894dbf0a484baa971e988b96db25: stopping tablet replica I0918 20:39:32.078874 19320 master.cc:258] Master@127.18.222.60:32967 shutdown complete. [ OK ] RemoteKsckTest.TestFilterOnNotabletTable (4013 ms) [ RUN ] RemoteKsckTest.TestNonMatchingTabletIdFilter I0918 20:39:32.124868 19320 test_util.cc:212] Using random seed: -711954266 I0918 20:39:32.125653 19320 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.18.222.62:44595,127.18.222.61:39745,127.18.222.60:42475 I0918 20:39:32.137866 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 163592us I0918 20:39:32.138216 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:32.138345 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:32.138530 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:32.138643 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:32.139678 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-0-root/instance: uuid: "66819bf7f855441ebe5d7a179952d404" format_stamp: "Formatted at 2019-09-18 20:39:32 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:32.148732 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.005s sys 0.003s I0918 20:39:32.150285 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:32.157335 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:32.157526 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-0-root uuid: "66819bf7f855441ebe5d7a179952d404" format_stamp: "Formatted at 2019-09-18 20:39:32 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:32.157826 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:32.229920 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:44595 I0918 20:39:32.231781 19320 webserver.cc:374] Webserver started at http://127.18.222.62:35011/ using document root and password file I0918 20:39:32.233544 27549 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:32.247817 27549 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:32.262096 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 163592us I0918 20:39:32.262583 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:32.262831 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:32.263104 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:32.263273 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:32.264645 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-1-root/instance: uuid: "e17636b7f64f46a191dcc1efc4745c7f" format_stamp: "Formatted at 2019-09-18 20:39:32 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:32.270110 27549 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:32.271963 27549 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:39745: Network error: Client connection negotiation failed: client connection to 127.18.222.61:39745: connect: Connection refused (error 111) I0918 20:39:32.276579 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.012s sys 0.000s I0918 20:39:32.278285 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:32.286114 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:32.286305 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-1-root uuid: "e17636b7f64f46a191dcc1efc4745c7f" format_stamp: "Formatted at 2019-09-18 20:39:32 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:32.286579 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:32.329427 27549 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } attempt: 1 W0918 20:39:32.331796 27549 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:39745: Network error: Client connection negotiation failed: client connection to 127.18.222.61:39745: connect: Connection refused (error 111) I0918 20:39:32.352221 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:39745 I0918 20:39:32.354163 19320 webserver.cc:374] Webserver started at http://127.18.222.61:43089/ using document root and password file I0918 20:39:32.355960 27606 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:32.361393 27606 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:32.369222 27549 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } attempt: 2 I0918 20:39:32.375576 27606 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:32.382831 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 163592us I0918 20:39:32.383400 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:32.383689 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:32.384057 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:32.384229 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:32.384944 27549 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:32.385404 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-2-root/instance: uuid: "78c10bfe8db6498b999e68f31d650f25" format_stamp: "Formatted at 2019-09-18 20:39:32 on dist-test-slave-dist-test-slave-85lk" W0918 20:39:32.387256 27549 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:42475: Network error: Client connection negotiation failed: client connection to 127.18.222.60:42475: connect: Connection refused (error 111) I0918 20:39:32.392791 27606 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:32.394649 27606 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:42475: Network error: Client connection negotiation failed: client connection to 127.18.222.60:42475: connect: Connection refused (error 111) I0918 20:39:32.398358 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.012s user 0.012s sys 0.000s I0918 20:39:32.400069 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:32.408285 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0918 20:39:32.408486 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-2-root uuid: "78c10bfe8db6498b999e68f31d650f25" format_stamp: "Formatted at 2019-09-18 20:39:32 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:32.408738 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:32.437234 27606 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } attempt: 1 I0918 20:39:32.437674 27549 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } attempt: 1 W0918 20:39:32.439658 27606 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:42475: Network error: Client connection negotiation failed: client connection to 127.18.222.60:42475: connect: Connection refused (error 111) W0918 20:39:32.439957 27549 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:42475: Network error: Client connection negotiation failed: client connection to 127.18.222.60:42475: connect: Connection refused (error 111) I0918 20:39:32.486450 27549 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } attempt: 2 W0918 20:39:32.489527 27549 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:42475: Network error: Client connection negotiation failed: client connection to 127.18.222.60:42475: connect: Connection refused (error 111) I0918 20:39:32.494279 27606 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } attempt: 2 W0918 20:39:32.496966 27606 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:42475: Network error: Client connection negotiation failed: client connection to 127.18.222.60:42475: connect: Connection refused (error 111) I0918 20:39:32.498440 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:42475 I0918 20:39:32.500324 19320 webserver.cc:374] Webserver started at http://127.18.222.60:38127/ using document root and password file I0918 20:39:32.501242 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0918 20:39:32.502238 27664 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:32.507673 27664 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:32.519455 27664 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:32.529666 27664 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:32.546788 27664 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25: Bootstrap starting. I0918 20:39:32.550014 27664 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25: No blocks or log segments found. Creating new log. I0918 20:39:32.553449 27664 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25: No bootstrap required, opened a new log I0918 20:39:32.555225 27664 raft_consensus.cc:340] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } I0918 20:39:32.555814 27664 raft_consensus.cc:366] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:32.556021 27664 raft_consensus.cc:705] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c10bfe8db6498b999e68f31d650f25, State: Initialized, Role: FOLLOWER I0918 20:39:32.556620 27664 consensus_queue.cc:229] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } I0918 20:39:32.558303 27664 sys_catalog.cc:340] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } } I0918 20:39:32.559067 27664 sys_catalog.cc:343] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:32.558542 27668 sys_catalog.cc:340] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } } I0918 20:39:32.559509 27668 sys_catalog.cc:343] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:32.560297 27664 sys_catalog.cc:428] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25 [sys.catalog]: configured and running, proceeding with master startup. W0918 20:39:32.563529 27669 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:32.563781 27669 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:32.578976 27549 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } attempt: 3 I0918 20:39:32.593462 27606 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } attempt: 3 I0918 20:39:32.595031 27549 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404: Bootstrap starting. I0918 20:39:32.598454 27549 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404: No blocks or log segments found. Creating new log. I0918 20:39:32.602283 27549 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404: No bootstrap required, opened a new log I0918 20:39:32.604605 27549 raft_consensus.cc:340] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } I0918 20:39:32.605221 27549 raft_consensus.cc:366] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:32.605473 27549 raft_consensus.cc:705] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66819bf7f855441ebe5d7a179952d404, State: Initialized, Role: FOLLOWER I0918 20:39:32.606098 27549 consensus_queue.cc:229] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } I0918 20:39:32.607684 27671 sys_catalog.cc:340] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } } I0918 20:39:32.608253 27671 sys_catalog.cc:343] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:32.608731 27549 sys_catalog.cc:340] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } } I0918 20:39:32.609314 27549 sys_catalog.cc:343] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:32.610428 27549 sys_catalog.cc:428] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:32.610374 27606 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f: Bootstrap starting. I0918 20:39:32.611606 27672 catalog_manager.cc:569] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0918 20:39:32.612048 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 I0918 20:39:32.613857 27606 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f: No blocks or log segments found. Creating new log. I0918 20:39:32.616942 27606 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f: No bootstrap required, opened a new log I0918 20:39:32.618539 27606 raft_consensus.cc:340] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } I0918 20:39:32.619027 27606 raft_consensus.cc:366] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:32.619175 27606 raft_consensus.cc:705] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e17636b7f64f46a191dcc1efc4745c7f, State: Initialized, Role: FOLLOWER I0918 20:39:32.619675 27606 consensus_queue.cc:229] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } I0918 20:39:32.621111 27606 sys_catalog.cc:340] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } } I0918 20:39:32.621635 27606 sys_catalog.cc:343] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:32.621342 27674 sys_catalog.cc:340] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } } I0918 20:39:32.622093 27674 sys_catalog.cc:343] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:32.622645 27606 sys_catalog.cc:428] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:32.623618 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 W0918 20:39:32.625632 27675 catalog_manager.cc:1155] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:32.625926 27675 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:32.638955 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 163592us I0918 20:39:32.639346 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:32.639511 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:32.639788 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:32.639919 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:32.641027 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-0-root/instance: uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" format_stamp: "Formatted at 2019-09-18 20:39:32 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:32.650948 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s I0918 20:39:32.652573 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:32.660426 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:32.660645 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-0-root uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" format_stamp: "Formatted at 2019-09-18 20:39:32 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:32.660982 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:32.696377 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:32.696547 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:32.737463 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:37883 I0918 20:39:32.739248 19320 webserver.cc:374] Webserver started at http://127.18.222.1:38583/ using document root and password file I0918 20:39:32.746836 27671 raft_consensus.cc:472] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:32.747427 27671 raft_consensus.cc:494] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } I0918 20:39:32.749176 27671 leader_election.cc:296] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e17636b7f64f46a191dcc1efc4745c7f (127.18.222.61:39745), 78c10bfe8db6498b999e68f31d650f25 (127.18.222.60:42475) I0918 20:39:32.750252 27583 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66819bf7f855441ebe5d7a179952d404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e17636b7f64f46a191dcc1efc4745c7f" is_pre_election: true I0918 20:39:32.750874 27641 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66819bf7f855441ebe5d7a179952d404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c10bfe8db6498b999e68f31d650f25" is_pre_election: true I0918 20:39:32.751338 27583 raft_consensus.cc:2350] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66819bf7f855441ebe5d7a179952d404 in term 0. I0918 20:39:32.752023 27641 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66819bf7f855441ebe5d7a179952d404 in term 0. I0918 20:39:32.752857 27498 leader_election.cc:310] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66819bf7f855441ebe5d7a179952d404, e17636b7f64f46a191dcc1efc4745c7f; no voters: I0918 20:39:32.753878 27671 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:32.754199 27671 raft_consensus.cc:472] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:32.754536 27671 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:32.762781 27671 raft_consensus.cc:494] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } I0918 20:39:32.765081 27740 heartbeater.cc:322] Connected to a master server at 127.18.222.60:42475 I0918 20:39:32.765615 27740 heartbeater.cc:406] Registering TS with master... I0918 20:39:32.765941 27583 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66819bf7f855441ebe5d7a179952d404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e17636b7f64f46a191dcc1efc4745c7f" I0918 20:39:32.766229 27740 heartbeater.cc:455] Master 127.18.222.60:42475 requested a full tablet report, sending... I0918 20:39:32.766619 27583 raft_consensus.cc:2890] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:32.769295 27631 ts_manager.cc:186] Registered new tserver with Master: bcc22735ee2b42f9ad9c86b862a5d5bb (127.18.222.1:37883) I0918 20:39:32.770867 27741 heartbeater.cc:322] Connected to a master server at 127.18.222.61:39745 I0918 20:39:32.771185 27741 heartbeater.cc:406] Registering TS with master... I0918 20:39:32.772064 27741 heartbeater.cc:455] Master 127.18.222.61:39745 requested a full tablet report, sending... I0918 20:39:32.774103 27641 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66819bf7f855441ebe5d7a179952d404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c10bfe8db6498b999e68f31d650f25" I0918 20:39:32.774766 27641 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:32.774884 27583 raft_consensus.cc:2350] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66819bf7f855441ebe5d7a179952d404 in term 1. I0918 20:39:32.776002 27498 leader_election.cc:310] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66819bf7f855441ebe5d7a179952d404, e17636b7f64f46a191dcc1efc4745c7f; no voters: I0918 20:39:32.776281 27573 ts_manager.cc:186] Registered new tserver with Master: bcc22735ee2b42f9ad9c86b862a5d5bb (127.18.222.1:37883) I0918 20:39:32.776991 27671 leader_election.cc:296] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [CANDIDATE]: Term 1 election: Requested vote from peers e17636b7f64f46a191dcc1efc4745c7f (127.18.222.61:39745), 78c10bfe8db6498b999e68f31d650f25 (127.18.222.60:42475) I0918 20:39:32.778098 27671 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:32.778394 27671 raft_consensus.cc:667] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [term 1 LEADER]: Becoming Leader. State: Replica: 66819bf7f855441ebe5d7a179952d404, State: Running, Role: LEADER I0918 20:39:32.778821 27671 consensus_queue.cc:206] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } I0918 20:39:32.785929 27641 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66819bf7f855441ebe5d7a179952d404 in term 1. I0918 20:39:32.796952 27747 sys_catalog.cc:340] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66819bf7f855441ebe5d7a179952d404. Latest consensus state: current_term: 1 leader_uuid: "66819bf7f855441ebe5d7a179952d404" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } } I0918 20:39:32.797529 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 163592us I0918 20:39:32.797966 27747 sys_catalog.cc:343] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [sys.catalog]: This master's current role is: LEADER I0918 20:39:32.798339 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:32.798624 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:32.798976 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:32.799204 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:32.800422 27749 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:39:32.800726 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-1-root/instance: uuid: "abd15c7255ab4b42aa73364c37ae964c" format_stamp: "Formatted at 2019-09-18 20:39:32 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:32.800843 27742 heartbeater.cc:322] Connected to a master server at 127.18.222.62:44595 I0918 20:39:32.801470 27742 heartbeater.cc:406] Registering TS with master... I0918 20:39:32.802078 27742 heartbeater.cc:455] Master 127.18.222.62:44595 requested a full tablet report, sending... I0918 20:39:32.803966 27749 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:39:32.804405 27516 ts_manager.cc:186] Registered new tserver with Master: bcc22735ee2b42f9ad9c86b862a5d5bb (127.18.222.1:37883) I0918 20:39:32.818852 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.016s user 0.014s sys 0.000s I0918 20:39:32.822221 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:32.833428 19320 fs_manager.cc:419] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s I0918 20:39:32.833762 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-1-root uuid: "abd15c7255ab4b42aa73364c37ae964c" format_stamp: "Formatted at 2019-09-18 20:39:32 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:32.834082 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:32.848287 27641 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25 [term 1 FOLLOWER]: Refusing update from remote peer 66819bf7f855441ebe5d7a179952d404: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:32.849859 27747 consensus_queue.cc:984] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:32.851557 27583 raft_consensus.cc:1184] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f [term 1 FOLLOWER]: Refusing update from remote peer 66819bf7f855441ebe5d7a179952d404: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:32.853169 27747 consensus_queue.cc:984] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:32.864235 27674 sys_catalog.cc:340] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66819bf7f855441ebe5d7a179952d404. Latest consensus state: current_term: 1 leader_uuid: "66819bf7f855441ebe5d7a179952d404" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } } I0918 20:39:32.865175 27674 sys_catalog.cc:343] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:32.867352 27668 sys_catalog.cc:340] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66819bf7f855441ebe5d7a179952d404. Latest consensus state: current_term: 1 leader_uuid: "66819bf7f855441ebe5d7a179952d404" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } } I0918 20:39:32.868286 27668 sys_catalog.cc:343] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:32.897457 27668 sys_catalog.cc:340] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66819bf7f855441ebe5d7a179952d404" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } } I0918 20:39:32.898460 27668 sys_catalog.cc:343] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:32.905467 27747 sys_catalog.cc:340] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66819bf7f855441ebe5d7a179952d404" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } } I0918 20:39:32.907047 27747 sys_catalog.cc:343] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [sys.catalog]: This master's current role is: LEADER I0918 20:39:32.904790 27671 sys_catalog.cc:340] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66819bf7f855441ebe5d7a179952d404" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } } I0918 20:39:32.910025 27674 sys_catalog.cc:340] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66819bf7f855441ebe5d7a179952d404" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66819bf7f855441ebe5d7a179952d404" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 44595 } } peers { permanent_uuid: "e17636b7f64f46a191dcc1efc4745c7f" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39745 } } peers { permanent_uuid: "78c10bfe8db6498b999e68f31d650f25" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42475 } } } I0918 20:39:32.911468 27674 sys_catalog.cc:343] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:32.910487 27671 sys_catalog.cc:343] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [sys.catalog]: This master's current role is: LEADER I0918 20:39:32.914454 27749 catalog_manager.cc:981] Generated new certificate authority record I0918 20:39:32.916401 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:32.916723 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:32.917435 27749 catalog_manager.cc:1106] Loading token signing keys... I0918 20:39:32.943583 27749 catalog_manager.cc:4446] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404: Generated new TSK 0 I0918 20:39:32.944438 27749 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:39:32.956180 27516 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49098 I0918 20:39:32.970340 27742 heartbeater.cc:447] Master 127.18.222.62:44595 was elected leader, sending a full tablet report... I0918 20:39:32.989966 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.2:32863 I0918 20:39:32.992272 19320 webserver.cc:374] Webserver started at http://127.18.222.2:45575/ using document root and password file I0918 20:39:33.024647 27823 heartbeater.cc:322] Connected to a master server at 127.18.222.60:42475 I0918 20:39:33.025025 27823 heartbeater.cc:406] Registering TS with master... I0918 20:39:33.025594 27823 heartbeater.cc:455] Master 127.18.222.60:42475 requested a full tablet report, sending... I0918 20:39:33.028815 27631 ts_manager.cc:186] Registered new tserver with Master: abd15c7255ab4b42aa73364c37ae964c (127.18.222.2:32863) I0918 20:39:33.035784 27827 heartbeater.cc:322] Connected to a master server at 127.18.222.62:44595 I0918 20:39:33.036175 27827 heartbeater.cc:406] Registering TS with master... I0918 20:39:33.036850 27827 heartbeater.cc:455] Master 127.18.222.62:44595 requested a full tablet report, sending... I0918 20:39:33.036857 27824 heartbeater.cc:322] Connected to a master server at 127.18.222.61:39745 I0918 20:39:33.037308 27824 heartbeater.cc:406] Registering TS with master... I0918 20:39:33.037773 27824 heartbeater.cc:455] Master 127.18.222.61:39745 requested a full tablet report, sending... I0918 20:39:33.038221 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 164092us I0918 20:39:33.038722 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:33.038944 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:33.038885 27516 ts_manager.cc:186] Registered new tserver with Master: abd15c7255ab4b42aa73364c37ae964c (127.18.222.2:32863) I0918 20:39:33.039443 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:33.039674 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:33.040676 27573 ts_manager.cc:186] Registered new tserver with Master: abd15c7255ab4b42aa73364c37ae964c (127.18.222.2:32863) I0918 20:39:33.041801 27516 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49104 I0918 20:39:33.043172 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-2-root/instance: uuid: "b5c1cf26d13945c781d097f0a1747619" format_stamp: "Formatted at 2019-09-18 20:39:33 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:33.058846 27827 heartbeater.cc:447] Master 127.18.222.62:44595 was elected leader, sending a full tablet report... I0918 20:39:33.058925 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.015s user 0.014s sys 0.000s I0918 20:39:33.062572 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:33.074018 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0918 20:39:33.074344 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-2-root uuid: "b5c1cf26d13945c781d097f0a1747619" format_stamp: "Formatted at 2019-09-18 20:39:33 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:33.074678 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:33.108294 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:33.108597 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:33.174568 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.3:36189 I0918 20:39:33.177670 19320 webserver.cc:374] Webserver started at http://127.18.222.3:41887/ using document root and password file I0918 20:39:33.276054 27894 heartbeater.cc:322] Connected to a master server at 127.18.222.60:42475 I0918 20:39:33.276486 27894 heartbeater.cc:406] Registering TS with master... I0918 20:39:33.277130 27894 heartbeater.cc:455] Master 127.18.222.60:42475 requested a full tablet report, sending... I0918 20:39:33.279681 27631 ts_manager.cc:186] Registered new tserver with Master: b5c1cf26d13945c781d097f0a1747619 (127.18.222.3:36189) I0918 20:39:33.287619 27895 heartbeater.cc:322] Connected to a master server at 127.18.222.61:39745 I0918 20:39:33.288141 27895 heartbeater.cc:406] Registering TS with master... I0918 20:39:33.288034 27896 heartbeater.cc:322] Connected to a master server at 127.18.222.62:44595 I0918 20:39:33.288712 27896 heartbeater.cc:406] Registering TS with master... I0918 20:39:33.288906 27895 heartbeater.cc:455] Master 127.18.222.61:39745 requested a full tablet report, sending... I0918 20:39:33.289295 27896 heartbeater.cc:455] Master 127.18.222.62:44595 requested a full tablet report, sending... I0918 20:39:33.291090 27516 ts_manager.cc:186] Registered new tserver with Master: b5c1cf26d13945c781d097f0a1747619 (127.18.222.3:36189) I0918 20:39:33.291633 27573 ts_manager.cc:186] Registered new tserver with Master: b5c1cf26d13945c781d097f0a1747619 (127.18.222.3:36189) I0918 20:39:33.293603 27516 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49110 I0918 20:39:33.293792 19320 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.029506105s I0918 20:39:33.308020 27896 heartbeater.cc:447] Master 127.18.222.62:44595 was elected leader, sending a full tablet report... I0918 20:39:33.373452 27515 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49112: name: "ksck-test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\n\000\000\000\004\001\024\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:39:33.375293 27515 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:39:33.500277 27707 tablet_service.cc:1022] Processing CreateTablet for tablet 8df7cac7256a47d2be58bd3e4103e7e6 (table=ksck-test-table [id=34fb39cca0d34a53917f57bd9c72cee5]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:33.501400 27707 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 8df7cac7256a47d2be58bd3e4103e7e6. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:33.509124 27706 tablet_service.cc:1022] Processing CreateTablet for tablet 737e3947c5f142588c2d231bc161d0c0 (table=ksck-test-table [id=34fb39cca0d34a53917f57bd9c72cee5]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:33.510422 27706 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 737e3947c5f142588c2d231bc161d0c0. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:33.512444 27790 tablet_service.cc:1022] Processing CreateTablet for tablet 8df7cac7256a47d2be58bd3e4103e7e6 (table=ksck-test-table [id=34fb39cca0d34a53917f57bd9c72cee5]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:33.513198 27789 tablet_service.cc:1022] Processing CreateTablet for tablet 737e3947c5f142588c2d231bc161d0c0 (table=ksck-test-table [id=34fb39cca0d34a53917f57bd9c72cee5]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:33.513550 27790 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 8df7cac7256a47d2be58bd3e4103e7e6. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:33.514227 27789 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 737e3947c5f142588c2d231bc161d0c0. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:33.516454 27788 tablet_service.cc:1022] Processing CreateTablet for tablet 990a87a7d5c245f39918d191a252b4b6 (table=ksck-test-table [id=34fb39cca0d34a53917f57bd9c72cee5]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:33.517750 27788 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 990a87a7d5c245f39918d191a252b4b6. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:33.520485 27705 tablet_service.cc:1022] Processing CreateTablet for tablet 990a87a7d5c245f39918d191a252b4b6 (table=ksck-test-table [id=34fb39cca0d34a53917f57bd9c72cee5]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:33.521596 27705 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 990a87a7d5c245f39918d191a252b4b6. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:33.558635 27861 tablet_service.cc:1022] Processing CreateTablet for tablet 8df7cac7256a47d2be58bd3e4103e7e6 (table=ksck-test-table [id=34fb39cca0d34a53917f57bd9c72cee5]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:33.560431 27861 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 8df7cac7256a47d2be58bd3e4103e7e6. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:33.560658 27859 tablet_service.cc:1022] Processing CreateTablet for tablet 990a87a7d5c245f39918d191a252b4b6 (table=ksck-test-table [id=34fb39cca0d34a53917f57bd9c72cee5]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:33.561736 27859 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 990a87a7d5c245f39918d191a252b4b6. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:33.563823 27860 tablet_service.cc:1022] Processing CreateTablet for tablet 737e3947c5f142588c2d231bc161d0c0 (table=ksck-test-table [id=34fb39cca0d34a53917f57bd9c72cee5]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:33.564987 27860 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 737e3947c5f142588c2d231bc161d0c0. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:33.585357 27915 tablet_bootstrap.cc:439] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c: Bootstrap starting. I0918 20:39:33.590294 27915 tablet_bootstrap.cc:590] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c: No blocks or log segments found. Creating new log. I0918 20:39:33.611510 27916 tablet_bootstrap.cc:439] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb: Bootstrap starting. I0918 20:39:33.611565 27669 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25: acquiring CA information for follower catalog manager: success I0918 20:39:33.626864 27916 tablet_bootstrap.cc:590] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb: No blocks or log segments found. Creating new log. I0918 20:39:33.652750 27675 catalog_manager.cc:1153] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f: acquiring CA information for follower catalog manager: success I0918 20:39:33.660393 27916 tablet_bootstrap.cc:439] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb: No bootstrap required, opened a new log I0918 20:39:33.661059 27916 ts_tablet_manager.cc:1085] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb: Time spent bootstrapping tablet: real 0.050s user 0.021s sys 0.004s I0918 20:39:33.659633 27669 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:33.663545 27916 raft_consensus.cc:340] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.664886 27916 raft_consensus.cc:366] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:33.665132 27916 raft_consensus.cc:705] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc22735ee2b42f9ad9c86b862a5d5bb, State: Initialized, Role: FOLLOWER I0918 20:39:33.665918 27916 consensus_queue.cc:229] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.670351 27915 tablet_bootstrap.cc:439] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c: No bootstrap required, opened a new log I0918 20:39:33.671139 27915 ts_tablet_manager.cc:1085] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.023s I0918 20:39:33.672899 27917 tablet_bootstrap.cc:439] T 737e3947c5f142588c2d231bc161d0c0 P b5c1cf26d13945c781d097f0a1747619: Bootstrap starting. I0918 20:39:33.673797 27915 raft_consensus.cc:340] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.674670 27915 raft_consensus.cc:366] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:33.674955 27915 raft_consensus.cc:705] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abd15c7255ab4b42aa73364c37ae964c, State: Initialized, Role: FOLLOWER I0918 20:39:33.675917 27915 consensus_queue.cc:229] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.686754 27916 ts_tablet_manager.cc:1113] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb: Time spent starting tablet: real 0.025s user 0.009s sys 0.004s I0918 20:39:33.689797 27917 tablet_bootstrap.cc:590] T 737e3947c5f142588c2d231bc161d0c0 P b5c1cf26d13945c781d097f0a1747619: No blocks or log segments found. Creating new log. I0918 20:39:33.682888 27675 catalog_manager.cc:1181] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:33.692953 27915 ts_tablet_manager.cc:1113] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c: Time spent starting tablet: real 0.021s user 0.004s sys 0.004s I0918 20:39:33.697206 27915 tablet_bootstrap.cc:439] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c: Bootstrap starting. I0918 20:39:33.700827 27915 tablet_bootstrap.cc:590] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c: No blocks or log segments found. Creating new log. I0918 20:39:33.703073 27916 tablet_bootstrap.cc:439] T 737e3947c5f142588c2d231bc161d0c0 P bcc22735ee2b42f9ad9c86b862a5d5bb: Bootstrap starting. I0918 20:39:33.707428 27916 tablet_bootstrap.cc:590] T 737e3947c5f142588c2d231bc161d0c0 P bcc22735ee2b42f9ad9c86b862a5d5bb: No blocks or log segments found. Creating new log. I0918 20:39:33.715353 27915 tablet_bootstrap.cc:439] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c: No bootstrap required, opened a new log I0918 20:39:33.715963 27915 ts_tablet_manager.cc:1085] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.001s I0918 20:39:33.717466 27917 tablet_bootstrap.cc:439] T 737e3947c5f142588c2d231bc161d0c0 P b5c1cf26d13945c781d097f0a1747619: No bootstrap required, opened a new log I0918 20:39:33.718070 27917 ts_tablet_manager.cc:1085] T 737e3947c5f142588c2d231bc161d0c0 P b5c1cf26d13945c781d097f0a1747619: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.004s I0918 20:39:33.717837 27915 raft_consensus.cc:340] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.718585 27915 raft_consensus.cc:366] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:33.718858 27915 raft_consensus.cc:705] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abd15c7255ab4b42aa73364c37ae964c, State: Initialized, Role: FOLLOWER I0918 20:39:33.719641 27915 consensus_queue.cc:229] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.724628 27915 ts_tablet_manager.cc:1113] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s I0918 20:39:33.725442 27915 tablet_bootstrap.cc:439] T 8df7cac7256a47d2be58bd3e4103e7e6 P abd15c7255ab4b42aa73364c37ae964c: Bootstrap starting. I0918 20:39:33.728624 27915 tablet_bootstrap.cc:590] T 8df7cac7256a47d2be58bd3e4103e7e6 P abd15c7255ab4b42aa73364c37ae964c: No blocks or log segments found. Creating new log. I0918 20:39:33.740805 27916 tablet_bootstrap.cc:439] T 737e3947c5f142588c2d231bc161d0c0 P bcc22735ee2b42f9ad9c86b862a5d5bb: No bootstrap required, opened a new log I0918 20:39:33.741420 27916 ts_tablet_manager.cc:1085] T 737e3947c5f142588c2d231bc161d0c0 P bcc22735ee2b42f9ad9c86b862a5d5bb: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.003s I0918 20:39:33.743453 27917 raft_consensus.cc:340] T 737e3947c5f142588c2d231bc161d0c0 P b5c1cf26d13945c781d097f0a1747619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.744185 27917 raft_consensus.cc:366] T 737e3947c5f142588c2d231bc161d0c0 P b5c1cf26d13945c781d097f0a1747619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:33.744578 27917 raft_consensus.cc:705] T 737e3947c5f142588c2d231bc161d0c0 P b5c1cf26d13945c781d097f0a1747619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c1cf26d13945c781d097f0a1747619, State: Initialized, Role: FOLLOWER I0918 20:39:33.745713 27916 raft_consensus.cc:340] T 737e3947c5f142588c2d231bc161d0c0 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.746703 27916 raft_consensus.cc:366] T 737e3947c5f142588c2d231bc161d0c0 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:33.747023 27916 raft_consensus.cc:705] T 737e3947c5f142588c2d231bc161d0c0 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc22735ee2b42f9ad9c86b862a5d5bb, State: Initialized, Role: FOLLOWER I0918 20:39:33.746073 27917 consensus_queue.cc:229] T 737e3947c5f142588c2d231bc161d0c0 P b5c1cf26d13945c781d097f0a1747619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.749949 27916 consensus_queue.cc:229] T 737e3947c5f142588c2d231bc161d0c0 P bcc22735ee2b42f9ad9c86b862a5d5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.754178 27916 ts_tablet_manager.cc:1113] T 737e3947c5f142588c2d231bc161d0c0 P bcc22735ee2b42f9ad9c86b862a5d5bb: Time spent starting tablet: real 0.012s user 0.006s sys 0.002s I0918 20:39:33.755141 27916 tablet_bootstrap.cc:439] T 990a87a7d5c245f39918d191a252b4b6 P bcc22735ee2b42f9ad9c86b862a5d5bb: Bootstrap starting. I0918 20:39:33.755833 27917 ts_tablet_manager.cc:1113] T 737e3947c5f142588c2d231bc161d0c0 P b5c1cf26d13945c781d097f0a1747619: Time spent starting tablet: real 0.037s user 0.008s sys 0.001s I0918 20:39:33.756884 27917 tablet_bootstrap.cc:439] T 8df7cac7256a47d2be58bd3e4103e7e6 P b5c1cf26d13945c781d097f0a1747619: Bootstrap starting. I0918 20:39:33.758424 27921 raft_consensus.cc:472] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:33.759176 27916 tablet_bootstrap.cc:590] T 990a87a7d5c245f39918d191a252b4b6 P bcc22735ee2b42f9ad9c86b862a5d5bb: No blocks or log segments found. Creating new log. I0918 20:39:33.759029 27921 raft_consensus.cc:494] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.764355 27921 leader_election.cc:296] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcc22735ee2b42f9ad9c86b862a5d5bb (127.18.222.1:37883), b5c1cf26d13945c781d097f0a1747619 (127.18.222.3:36189) I0918 20:39:33.747107 27915 tablet_bootstrap.cc:439] T 8df7cac7256a47d2be58bd3e4103e7e6 P abd15c7255ab4b42aa73364c37ae964c: No bootstrap required, opened a new log I0918 20:39:33.760565 27917 tablet_bootstrap.cc:590] T 8df7cac7256a47d2be58bd3e4103e7e6 P b5c1cf26d13945c781d097f0a1747619: No blocks or log segments found. Creating new log. I0918 20:39:33.808339 27915 ts_tablet_manager.cc:1085] T 8df7cac7256a47d2be58bd3e4103e7e6 P abd15c7255ab4b42aa73364c37ae964c: Time spent bootstrapping tablet: real 0.083s user 0.006s sys 0.003s I0918 20:39:33.813094 27916 tablet_bootstrap.cc:439] T 990a87a7d5c245f39918d191a252b4b6 P bcc22735ee2b42f9ad9c86b862a5d5bb: No bootstrap required, opened a new log I0918 20:39:33.816210 27916 ts_tablet_manager.cc:1085] T 990a87a7d5c245f39918d191a252b4b6 P bcc22735ee2b42f9ad9c86b862a5d5bb: Time spent bootstrapping tablet: real 0.061s user 0.023s sys 0.029s I0918 20:39:33.824667 27915 raft_consensus.cc:340] T 8df7cac7256a47d2be58bd3e4103e7e6 P abd15c7255ab4b42aa73364c37ae964c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.827127 27917 tablet_bootstrap.cc:439] T 8df7cac7256a47d2be58bd3e4103e7e6 P b5c1cf26d13945c781d097f0a1747619: No bootstrap required, opened a new log I0918 20:39:33.828217 27917 ts_tablet_manager.cc:1085] T 8df7cac7256a47d2be58bd3e4103e7e6 P b5c1cf26d13945c781d097f0a1747619: Time spent bootstrapping tablet: real 0.072s user 0.006s sys 0.001s I0918 20:39:33.823278 27916 raft_consensus.cc:340] T 990a87a7d5c245f39918d191a252b4b6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.830515 27916 raft_consensus.cc:366] T 990a87a7d5c245f39918d191a252b4b6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:33.827549 27915 raft_consensus.cc:366] T 8df7cac7256a47d2be58bd3e4103e7e6 P abd15c7255ab4b42aa73364c37ae964c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:33.830811 27916 raft_consensus.cc:705] T 990a87a7d5c245f39918d191a252b4b6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc22735ee2b42f9ad9c86b862a5d5bb, State: Initialized, Role: FOLLOWER I0918 20:39:33.830586 27917 raft_consensus.cc:340] T 8df7cac7256a47d2be58bd3e4103e7e6 P b5c1cf26d13945c781d097f0a1747619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.835525 27915 raft_consensus.cc:705] T 8df7cac7256a47d2be58bd3e4103e7e6 P abd15c7255ab4b42aa73364c37ae964c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abd15c7255ab4b42aa73364c37ae964c, State: Initialized, Role: FOLLOWER I0918 20:39:33.835309 27916 consensus_queue.cc:229] T 990a87a7d5c245f39918d191a252b4b6 P bcc22735ee2b42f9ad9c86b862a5d5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.836568 27915 consensus_queue.cc:229] T 8df7cac7256a47d2be58bd3e4103e7e6 P abd15c7255ab4b42aa73364c37ae964c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.838876 27916 ts_tablet_manager.cc:1113] T 990a87a7d5c245f39918d191a252b4b6 P bcc22735ee2b42f9ad9c86b862a5d5bb: Time spent starting tablet: real 0.018s user 0.004s sys 0.004s I0918 20:39:33.857533 27915 ts_tablet_manager.cc:1113] T 8df7cac7256a47d2be58bd3e4103e7e6 P abd15c7255ab4b42aa73364c37ae964c: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s I0918 20:39:33.835445 27917 raft_consensus.cc:366] T 8df7cac7256a47d2be58bd3e4103e7e6 P b5c1cf26d13945c781d097f0a1747619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:33.858762 27917 raft_consensus.cc:705] T 8df7cac7256a47d2be58bd3e4103e7e6 P b5c1cf26d13945c781d097f0a1747619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c1cf26d13945c781d097f0a1747619, State: Initialized, Role: FOLLOWER I0918 20:39:33.859679 27917 consensus_queue.cc:229] T 8df7cac7256a47d2be58bd3e4103e7e6 P b5c1cf26d13945c781d097f0a1747619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.864722 27917 ts_tablet_manager.cc:1113] T 8df7cac7256a47d2be58bd3e4103e7e6 P b5c1cf26d13945c781d097f0a1747619: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s I0918 20:39:33.867936 27917 tablet_bootstrap.cc:439] T 990a87a7d5c245f39918d191a252b4b6 P b5c1cf26d13945c781d097f0a1747619: Bootstrap starting. I0918 20:39:33.874657 27871 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "990a87a7d5c245f39918d191a252b4b6" candidate_uuid: "abd15c7255ab4b42aa73364c37ae964c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c1cf26d13945c781d097f0a1747619" is_pre_election: true I0918 20:39:33.876870 27917 tablet_bootstrap.cc:590] T 990a87a7d5c245f39918d191a252b4b6 P b5c1cf26d13945c781d097f0a1747619: No blocks or log segments found. Creating new log. W0918 20:39:33.877790 27752 leader_election.cc:348] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5c1cf26d13945c781d097f0a1747619 (127.18.222.3:36189): Illegal state: must be running to vote when last-logged opid is not known I0918 20:39:33.885521 27717 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "990a87a7d5c245f39918d191a252b4b6" candidate_uuid: "abd15c7255ab4b42aa73364c37ae964c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" is_pre_election: true I0918 20:39:33.886391 27717 raft_consensus.cc:2350] T 990a87a7d5c245f39918d191a252b4b6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abd15c7255ab4b42aa73364c37ae964c in term 0. I0918 20:39:33.887858 27754 leader_election.cc:310] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: abd15c7255ab4b42aa73364c37ae964c, bcc22735ee2b42f9ad9c86b862a5d5bb; no voters: b5c1cf26d13945c781d097f0a1747619 I0918 20:39:33.889199 27921 raft_consensus.cc:2644] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:33.889470 27921 raft_consensus.cc:472] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:33.889760 27921 raft_consensus.cc:2890] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:33.890048 27932 raft_consensus.cc:472] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:33.890285 27917 tablet_bootstrap.cc:439] T 990a87a7d5c245f39918d191a252b4b6 P b5c1cf26d13945c781d097f0a1747619: No bootstrap required, opened a new log I0918 20:39:33.890594 27932 raft_consensus.cc:494] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.893206 27717 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "737e3947c5f142588c2d231bc161d0c0" candidate_uuid: "abd15c7255ab4b42aa73364c37ae964c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" is_pre_election: true I0918 20:39:33.893693 27871 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "737e3947c5f142588c2d231bc161d0c0" candidate_uuid: "abd15c7255ab4b42aa73364c37ae964c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c1cf26d13945c781d097f0a1747619" is_pre_election: true I0918 20:39:33.894459 27871 raft_consensus.cc:2350] T 737e3947c5f142588c2d231bc161d0c0 P b5c1cf26d13945c781d097f0a1747619 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abd15c7255ab4b42aa73364c37ae964c in term 0. I0918 20:39:33.893859 27717 raft_consensus.cc:2350] T 737e3947c5f142588c2d231bc161d0c0 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abd15c7255ab4b42aa73364c37ae964c in term 0. I0918 20:39:33.895366 27752 leader_election.cc:310] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abd15c7255ab4b42aa73364c37ae964c, b5c1cf26d13945c781d097f0a1747619; no voters: I0918 20:39:33.895634 27932 leader_election.cc:296] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcc22735ee2b42f9ad9c86b862a5d5bb (127.18.222.1:37883), b5c1cf26d13945c781d097f0a1747619 (127.18.222.3:36189) I0918 20:39:33.896307 27932 raft_consensus.cc:2644] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:33.897042 27917 ts_tablet_manager.cc:1085] T 990a87a7d5c245f39918d191a252b4b6 P b5c1cf26d13945c781d097f0a1747619: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.003s I0918 20:39:33.901659 27921 raft_consensus.cc:494] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.906044 27932 raft_consensus.cc:472] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:33.907168 27932 raft_consensus.cc:2890] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:33.910519 27921 leader_election.cc:296] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [CANDIDATE]: Term 1 election: Requested vote from peers bcc22735ee2b42f9ad9c86b862a5d5bb (127.18.222.1:37883), b5c1cf26d13945c781d097f0a1747619 (127.18.222.3:36189) I0918 20:39:33.918918 27717 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "990a87a7d5c245f39918d191a252b4b6" candidate_uuid: "abd15c7255ab4b42aa73364c37ae964c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" I0918 20:39:33.921851 27717 raft_consensus.cc:2890] T 990a87a7d5c245f39918d191a252b4b6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:33.929651 27871 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "990a87a7d5c245f39918d191a252b4b6" candidate_uuid: "abd15c7255ab4b42aa73364c37ae964c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c1cf26d13945c781d097f0a1747619" I0918 20:39:33.930436 27717 raft_consensus.cc:2350] T 990a87a7d5c245f39918d191a252b4b6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abd15c7255ab4b42aa73364c37ae964c in term 1. W0918 20:39:33.939102 27752 leader_election.cc:348] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b5c1cf26d13945c781d097f0a1747619 (127.18.222.3:36189): Illegal state: must be running to vote when last-logged opid is not known I0918 20:39:33.941720 27754 leader_election.cc:310] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: abd15c7255ab4b42aa73364c37ae964c, bcc22735ee2b42f9ad9c86b862a5d5bb; no voters: b5c1cf26d13945c781d097f0a1747619 I0918 20:39:33.941713 27917 raft_consensus.cc:340] T 990a87a7d5c245f39918d191a252b4b6 P b5c1cf26d13945c781d097f0a1747619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.942800 27917 raft_consensus.cc:366] T 990a87a7d5c245f39918d191a252b4b6 P b5c1cf26d13945c781d097f0a1747619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:33.943122 27917 raft_consensus.cc:705] T 990a87a7d5c245f39918d191a252b4b6 P b5c1cf26d13945c781d097f0a1747619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c1cf26d13945c781d097f0a1747619, State: Initialized, Role: FOLLOWER I0918 20:39:33.943199 27933 raft_consensus.cc:2644] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:33.944367 27933 raft_consensus.cc:667] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [term 1 LEADER]: Becoming Leader. State: Replica: abd15c7255ab4b42aa73364c37ae964c, State: Running, Role: LEADER I0918 20:39:33.944015 27917 consensus_queue.cc:229] T 990a87a7d5c245f39918d191a252b4b6 P b5c1cf26d13945c781d097f0a1747619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.945116 27933 consensus_queue.cc:206] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.946792 27917 ts_tablet_manager.cc:1113] T 990a87a7d5c245f39918d191a252b4b6 P b5c1cf26d13945c781d097f0a1747619: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s I0918 20:39:33.942912 27932 raft_consensus.cc:494] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.950117 27871 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "737e3947c5f142588c2d231bc161d0c0" candidate_uuid: "abd15c7255ab4b42aa73364c37ae964c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c1cf26d13945c781d097f0a1747619" I0918 20:39:33.950568 27871 raft_consensus.cc:2890] T 737e3947c5f142588c2d231bc161d0c0 P b5c1cf26d13945c781d097f0a1747619 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:33.957393 27871 raft_consensus.cc:2350] T 737e3947c5f142588c2d231bc161d0c0 P b5c1cf26d13945c781d097f0a1747619 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abd15c7255ab4b42aa73364c37ae964c in term 1. I0918 20:39:33.958771 27717 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "737e3947c5f142588c2d231bc161d0c0" candidate_uuid: "abd15c7255ab4b42aa73364c37ae964c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" I0918 20:39:33.959132 27752 leader_election.cc:310] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abd15c7255ab4b42aa73364c37ae964c, b5c1cf26d13945c781d097f0a1747619; no voters: I0918 20:39:33.959525 27717 raft_consensus.cc:2890] T 737e3947c5f142588c2d231bc161d0c0 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:33.960290 27921 raft_consensus.cc:2644] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:33.958700 27932 leader_election.cc:296] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c [CANDIDATE]: Term 1 election: Requested vote from peers bcc22735ee2b42f9ad9c86b862a5d5bb (127.18.222.1:37883), b5c1cf26d13945c781d097f0a1747619 (127.18.222.3:36189) I0918 20:39:33.961833 27921 raft_consensus.cc:667] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c [term 1 LEADER]: Becoming Leader. State: Replica: abd15c7255ab4b42aa73364c37ae964c, State: Running, Role: LEADER I0918 20:39:33.963059 27921 consensus_queue.cc:206] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:33.997941 27516 catalog_manager.cc:4131] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c reported cstate change: term changed from 0 to 1, leader changed from to abd15c7255ab4b42aa73364c37ae964c (127.18.222.2). New cstate: current_term: 1 leader_uuid: "abd15c7255ab4b42aa73364c37ae964c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } health_report { overall_health: UNKNOWN } } } I0918 20:39:33.998523 27717 raft_consensus.cc:2350] T 737e3947c5f142588c2d231bc161d0c0 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abd15c7255ab4b42aa73364c37ae964c in term 1. I0918 20:39:34.046960 27515 catalog_manager.cc:4131] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c reported cstate change: term changed from 0 to 1, leader changed from to abd15c7255ab4b42aa73364c37ae964c (127.18.222.2). New cstate: current_term: 1 leader_uuid: "abd15c7255ab4b42aa73364c37ae964c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } health_report { overall_health: UNKNOWN } } } I0918 20:39:34.114945 27920 raft_consensus.cc:472] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:34.115396 27920 raft_consensus.cc:494] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:34.117491 27920 leader_election.cc:296] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abd15c7255ab4b42aa73364c37ae964c (127.18.222.2:32863), b5c1cf26d13945c781d097f0a1747619 (127.18.222.3:36189) I0918 20:39:34.151011 27800 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8df7cac7256a47d2be58bd3e4103e7e6" candidate_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abd15c7255ab4b42aa73364c37ae964c" is_pre_election: true I0918 20:39:34.151824 27800 raft_consensus.cc:2350] T 8df7cac7256a47d2be58bd3e4103e7e6 P abd15c7255ab4b42aa73364c37ae964c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcc22735ee2b42f9ad9c86b862a5d5bb in term 0. I0918 20:39:34.153669 27679 leader_election.cc:310] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abd15c7255ab4b42aa73364c37ae964c, bcc22735ee2b42f9ad9c86b862a5d5bb; no voters: I0918 20:39:34.154496 27920 raft_consensus.cc:2644] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:34.154806 27920 raft_consensus.cc:472] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:34.155063 27920 raft_consensus.cc:2890] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:34.157730 27871 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8df7cac7256a47d2be58bd3e4103e7e6" candidate_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c1cf26d13945c781d097f0a1747619" is_pre_election: true I0918 20:39:34.158623 27871 raft_consensus.cc:2350] T 8df7cac7256a47d2be58bd3e4103e7e6 P b5c1cf26d13945c781d097f0a1747619 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcc22735ee2b42f9ad9c86b862a5d5bb in term 0. I0918 20:39:34.165402 27920 raft_consensus.cc:494] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:34.167356 27920 leader_election.cc:296] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb [CANDIDATE]: Term 1 election: Requested vote from peers abd15c7255ab4b42aa73364c37ae964c (127.18.222.2:32863), b5c1cf26d13945c781d097f0a1747619 (127.18.222.3:36189) I0918 20:39:34.168179 27800 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8df7cac7256a47d2be58bd3e4103e7e6" candidate_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abd15c7255ab4b42aa73364c37ae964c" I0918 20:39:34.168411 27871 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8df7cac7256a47d2be58bd3e4103e7e6" candidate_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c1cf26d13945c781d097f0a1747619" I0918 20:39:34.168912 27800 raft_consensus.cc:2890] T 8df7cac7256a47d2be58bd3e4103e7e6 P abd15c7255ab4b42aa73364c37ae964c [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:34.169020 27871 raft_consensus.cc:2890] T 8df7cac7256a47d2be58bd3e4103e7e6 P b5c1cf26d13945c781d097f0a1747619 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:34.175580 27871 raft_consensus.cc:2350] T 8df7cac7256a47d2be58bd3e4103e7e6 P b5c1cf26d13945c781d097f0a1747619 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcc22735ee2b42f9ad9c86b862a5d5bb in term 1. I0918 20:39:34.176892 27678 leader_election.cc:310] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5c1cf26d13945c781d097f0a1747619, bcc22735ee2b42f9ad9c86b862a5d5bb; no voters: I0918 20:39:34.178033 27920 raft_consensus.cc:2644] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:34.180948 27800 raft_consensus.cc:2350] T 8df7cac7256a47d2be58bd3e4103e7e6 P abd15c7255ab4b42aa73364c37ae964c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcc22735ee2b42f9ad9c86b862a5d5bb in term 1. I0918 20:39:34.182407 27920 raft_consensus.cc:667] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 1 LEADER]: Becoming Leader. State: Replica: bcc22735ee2b42f9ad9c86b862a5d5bb, State: Running, Role: LEADER I0918 20:39:34.183104 27920 consensus_queue.cc:206] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:34.193377 27516 catalog_manager.cc:4131] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb reported cstate change: term changed from 0 to 1, leader changed from to bcc22735ee2b42f9ad9c86b862a5d5bb (127.18.222.1). New cstate: current_term: 1 leader_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } health_report { overall_health: UNKNOWN } } } I0918 20:39:34.277670 27926 raft_consensus.cc:472] T 990a87a7d5c245f39918d191a252b4b6 P b5c1cf26d13945c781d097f0a1747619 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:34.278401 27926 raft_consensus.cc:494] T 990a87a7d5c245f39918d191a252b4b6 P b5c1cf26d13945c781d097f0a1747619 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 } } peers { permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 } } peers { permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 } } I0918 20:39:34.281469 27926 leader_election.cc:296] T 990a87a7d5c245f39918d191a252b4b6 P b5c1cf26d13945c781d097f0a1747619 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcc22735ee2b42f9ad9c86b862a5d5bb (127.18.222.1:37883), abd15c7255ab4b42aa73364c37ae964c (127.18.222.2:32863) I0918 20:39:34.385567 27800 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "990a87a7d5c245f39918d191a252b4b6" candidate_uuid: "b5c1cf26d13945c781d097f0a1747619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abd15c7255ab4b42aa73364c37ae964c" is_pre_election: true I0918 20:39:34.386433 27800 raft_consensus.cc:2311] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate b5c1cf26d13945c781d097f0a1747619 for term 1 because replica is either leader or believes a valid leader to be alive. I0918 20:39:34.415818 27717 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "990a87a7d5c245f39918d191a252b4b6" candidate_uuid: "b5c1cf26d13945c781d097f0a1747619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" is_pre_election: true I0918 20:39:34.416743 27717 raft_consensus.cc:2280] T 990a87a7d5c245f39918d191a252b4b6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b5c1cf26d13945c781d097f0a1747619 in current term 1: Already voted for candidate abd15c7255ab4b42aa73364c37ae964c in this term. I0918 20:39:34.417906 27833 leader_election.cc:310] T 990a87a7d5c245f39918d191a252b4b6 P b5c1cf26d13945c781d097f0a1747619 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5c1cf26d13945c781d097f0a1747619; no voters: abd15c7255ab4b42aa73364c37ae964c, bcc22735ee2b42f9ad9c86b862a5d5bb I0918 20:39:34.419529 27926 raft_consensus.cc:2890] T 990a87a7d5c245f39918d191a252b4b6 P b5c1cf26d13945c781d097f0a1747619 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:34.429144 27926 raft_consensus.cc:2592] T 990a87a7d5c245f39918d191a252b4b6 P b5c1cf26d13945c781d097f0a1747619 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0918 20:39:34.432564 27921 consensus_queue.cc:984] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:39:34.497109 27933 consensus_queue.cc:984] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcc22735ee2b42f9ad9c86b862a5d5bb" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 37883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:34.558725 27969 consensus_queue.cc:984] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:39:34.598820 27920 consensus_queue.cc:984] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:34.776566 27920 consensus_queue.cc:984] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "abd15c7255ab4b42aa73364c37ae964c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 32863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:34.777184 27933 consensus_queue.cc:984] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c1cf26d13945c781d097f0a1747619" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 36189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:39:35.235590 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:35.305688 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:35.306004 19320 tablet_replica.cc:273] T 990a87a7d5c245f39918d191a252b4b6 P bcc22735ee2b42f9ad9c86b862a5d5bb: stopping tablet replica I0918 20:39:35.306391 19320 raft_consensus.cc:2147] T 990a87a7d5c245f39918d191a252b4b6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:35.306702 19320 raft_consensus.cc:2174] T 990a87a7d5c245f39918d191a252b4b6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:35.310083 19320 tablet_replica.cc:273] T 737e3947c5f142588c2d231bc161d0c0 P bcc22735ee2b42f9ad9c86b862a5d5bb: stopping tablet replica I0918 20:39:35.310624 19320 raft_consensus.cc:2147] T 737e3947c5f142588c2d231bc161d0c0 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:35.310995 19320 raft_consensus.cc:2174] T 737e3947c5f142588c2d231bc161d0c0 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:35.317333 19320 tablet_replica.cc:273] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb: stopping tablet replica I0918 20:39:35.317869 19320 raft_consensus.cc:2147] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:35.318759 19320 raft_consensus.cc:2174] T 8df7cac7256a47d2be58bd3e4103e7e6 P bcc22735ee2b42f9ad9c86b862a5d5bb [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:35.339010 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:35.339231 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:35.339566 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:35.353130 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:35.400948 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:35.401307 19320 tablet_replica.cc:273] T 8df7cac7256a47d2be58bd3e4103e7e6 P abd15c7255ab4b42aa73364c37ae964c: stopping tablet replica I0918 20:39:35.401688 19320 raft_consensus.cc:2147] T 8df7cac7256a47d2be58bd3e4103e7e6 P abd15c7255ab4b42aa73364c37ae964c [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:35.401996 19320 raft_consensus.cc:2174] T 8df7cac7256a47d2be58bd3e4103e7e6 P abd15c7255ab4b42aa73364c37ae964c [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:35.404697 19320 tablet_replica.cc:273] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c: stopping tablet replica I0918 20:39:35.405082 19320 raft_consensus.cc:2147] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:35.405730 19320 raft_consensus.cc:2174] T 737e3947c5f142588c2d231bc161d0c0 P abd15c7255ab4b42aa73364c37ae964c [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:35.408802 19320 tablet_replica.cc:273] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c: stopping tablet replica I0918 20:39:35.409421 19320 raft_consensus.cc:2147] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:35.410439 19320 raft_consensus.cc:2174] T 990a87a7d5c245f39918d191a252b4b6 P abd15c7255ab4b42aa73364c37ae964c [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:35.428697 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:35.428902 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:35.429216 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:35.440337 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:35.470708 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:35.471037 19320 tablet_replica.cc:273] T 990a87a7d5c245f39918d191a252b4b6 P b5c1cf26d13945c781d097f0a1747619: stopping tablet replica I0918 20:39:35.471423 19320 raft_consensus.cc:2147] T 990a87a7d5c245f39918d191a252b4b6 P b5c1cf26d13945c781d097f0a1747619 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:35.471772 19320 raft_consensus.cc:2174] T 990a87a7d5c245f39918d191a252b4b6 P b5c1cf26d13945c781d097f0a1747619 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:35.474407 19320 tablet_replica.cc:273] T 8df7cac7256a47d2be58bd3e4103e7e6 P b5c1cf26d13945c781d097f0a1747619: stopping tablet replica I0918 20:39:35.474774 19320 raft_consensus.cc:2147] T 8df7cac7256a47d2be58bd3e4103e7e6 P b5c1cf26d13945c781d097f0a1747619 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:35.475091 19320 raft_consensus.cc:2174] T 8df7cac7256a47d2be58bd3e4103e7e6 P b5c1cf26d13945c781d097f0a1747619 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:35.477502 19320 tablet_replica.cc:273] T 737e3947c5f142588c2d231bc161d0c0 P b5c1cf26d13945c781d097f0a1747619: stopping tablet replica I0918 20:39:35.477885 19320 raft_consensus.cc:2147] T 737e3947c5f142588c2d231bc161d0c0 P b5c1cf26d13945c781d097f0a1747619 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:35.478161 19320 raft_consensus.cc:2174] T 737e3947c5f142588c2d231bc161d0c0 P b5c1cf26d13945c781d097f0a1747619 [term 1 FOLLOWER]: Raft consensus is shut down! W0918 20:39:35.822103 27836 debug-util.cc:397] Leaking SignalData structure 0x7b08002321a0 after lost signal to thread 19323 W0918 20:39:36.497732 19320 thread.cc:499] Waited for 1000ms trying to join with diag-logger (tid 27836) W0918 20:39:37.498159 19320 thread.cc:499] Waited for 2000ms trying to join with diag-logger (tid 27836) W0918 20:39:38.498564 19320 thread.cc:499] Waited for 3000ms trying to join with diag-logger (tid 27836) I0918 20:39:38.602476 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:38.602685 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:38.603102 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:38.616451 19320 master.cc:247] Master@127.18.222.62:44595 shutting down... I0918 20:39:38.635759 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:38.636529 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:38.636791 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 66819bf7f855441ebe5d7a179952d404: stopping tablet replica I0918 20:39:38.656024 19320 master.cc:258] Master@127.18.222.62:44595 shutdown complete. I0918 20:39:38.666138 19320 master.cc:247] Master@127.18.222.61:39745 shutting down... I0918 20:39:38.684146 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:38.684558 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:38.684789 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P e17636b7f64f46a191dcc1efc4745c7f: stopping tablet replica I0918 20:39:38.702780 19320 master.cc:258] Master@127.18.222.61:39745 shutdown complete. I0918 20:39:38.711941 19320 master.cc:247] Master@127.18.222.60:42475 shutting down... I0918 20:39:38.731501 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:38.731941 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:38.732165 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 78c10bfe8db6498b999e68f31d650f25: stopping tablet replica I0918 20:39:38.749958 19320 master.cc:258] Master@127.18.222.60:42475 shutdown complete. [ OK ] RemoteKsckTest.TestNonMatchingTabletIdFilter (6656 ms) [ RUN ] RemoteKsckTest.TestMatchingTabletIdFilter I0918 20:39:38.780848 19320 test_util.cc:212] Using random seed: -705298284 I0918 20:39:38.781622 19320 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.18.222.62:37137,127.18.222.61:38767,127.18.222.60:44875 I0918 20:39:38.794536 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 166592us I0918 20:39:38.794924 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:38.795097 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:38.795325 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:38.795454 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:38.796545 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-0-root/instance: uuid: "36211758fed84e26a80a3f1cd3c72444" format_stamp: "Formatted at 2019-09-18 20:39:38 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:38.805274 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.005s sys 0.003s I0918 20:39:38.806833 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:38.813839 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:39:38.814047 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-0-root uuid: "36211758fed84e26a80a3f1cd3c72444" format_stamp: "Formatted at 2019-09-18 20:39:38 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:38.814339 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:38.890924 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:37137 I0918 20:39:38.892724 19320 webserver.cc:374] Webserver started at http://127.18.222.62:35783/ using document root and password file I0918 20:39:38.894506 28057 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:38.899591 28057 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:38.909932 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 166592us I0918 20:39:38.910334 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:38.910526 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:38.910768 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:38.910898 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:38.912026 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-1-root/instance: uuid: "e7d0a1df3aa2499bb006d06bfbc57531" format_stamp: "Formatted at 2019-09-18 20:39:38 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:38.914897 28057 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:38.916733 28057 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:38767: Network error: Client connection negotiation failed: client connection to 127.18.222.61:38767: connect: Connection refused (error 111) I0918 20:39:38.922358 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.010s sys 0.000s I0918 20:39:38.924072 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:38.931881 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:39:38.932088 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-1-root uuid: "e7d0a1df3aa2499bb006d06bfbc57531" format_stamp: "Formatted at 2019-09-18 20:39:38 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:38.932337 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:38.935199 28057 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } attempt: 1 W0918 20:39:38.937789 28057 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:38767: Network error: Client connection negotiation failed: client connection to 127.18.222.61:38767: connect: Connection refused (error 111) I0918 20:39:38.991246 28057 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } attempt: 2 W0918 20:39:38.993436 28057 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:38767: Network error: Client connection negotiation failed: client connection to 127.18.222.61:38767: connect: Connection refused (error 111) I0918 20:39:39.032109 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:38767 I0918 20:39:39.034039 19320 webserver.cc:374] Webserver started at http://127.18.222.61:41669/ using document root and password file I0918 20:39:39.035872 28114 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:39.040982 28114 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:39.054495 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 167092us I0918 20:39:39.054549 28114 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:39.054864 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:39.055058 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:39.055284 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:39.055418 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:39.056830 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-2-root/instance: uuid: "80e9151b52aa4ba6bed4e6336b24a35e" format_stamp: "Formatted at 2019-09-18 20:39:39 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:39.068866 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.010s sys 0.001s I0918 20:39:39.070834 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:39.070806 28114 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:39.072839 28114 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:44875: Network error: Client connection negotiation failed: client connection to 127.18.222.60:44875: connect: Connection refused (error 111) I0918 20:39:39.077898 28057 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } attempt: 3 I0918 20:39:39.079520 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:39:39.080180 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-2-root uuid: "80e9151b52aa4ba6bed4e6336b24a35e" format_stamp: "Formatted at 2019-09-18 20:39:39 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:39.080523 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:39.090000 28057 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:39.091987 28057 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:44875: Network error: Client connection negotiation failed: client connection to 127.18.222.60:44875: connect: Connection refused (error 111) I0918 20:39:39.101272 28114 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } attempt: 1 W0918 20:39:39.103353 28114 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:44875: Network error: Client connection negotiation failed: client connection to 127.18.222.60:44875: connect: Connection refused (error 111) I0918 20:39:39.120442 28057 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } attempt: 1 W0918 20:39:39.122736 28057 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:44875: Network error: Client connection negotiation failed: client connection to 127.18.222.60:44875: connect: Connection refused (error 111) I0918 20:39:39.160220 28057 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } attempt: 2 W0918 20:39:39.162454 28057 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:44875: Network error: Client connection negotiation failed: client connection to 127.18.222.60:44875: connect: Connection refused (error 111) I0918 20:39:39.180214 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:44875 I0918 20:39:39.182662 19320 webserver.cc:374] Webserver started at http://127.18.222.60:40553/ using document root and password file I0918 20:39:39.183881 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0918 20:39:39.184768 28171 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:39.184813 28114 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } attempt: 2 I0918 20:39:39.190384 28171 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:39.207639 28114 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531: Bootstrap starting. I0918 20:39:39.208874 28171 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:39.211017 28114 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531: No blocks or log segments found. Creating new log. I0918 20:39:39.214817 28114 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531: No bootstrap required, opened a new log I0918 20:39:39.217411 28114 raft_consensus.cc:340] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } I0918 20:39:39.218232 28114 raft_consensus.cc:366] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:39.218513 28114 raft_consensus.cc:705] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7d0a1df3aa2499bb006d06bfbc57531, State: Initialized, Role: FOLLOWER I0918 20:39:39.219220 28114 consensus_queue.cc:229] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } I0918 20:39:39.220995 28171 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:39.221566 28114 sys_catalog.cc:340] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } } I0918 20:39:39.221724 28175 sys_catalog.cc:340] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } } I0918 20:39:39.222506 28114 sys_catalog.cc:343] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:39.222698 28175 sys_catalog.cc:343] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:39.224323 28114 sys_catalog.cc:428] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [sys.catalog]: configured and running, proceeding with master startup. W0918 20:39:39.227489 28176 catalog_manager.cc:1155] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:39.227747 28176 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:39.227959 28057 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } attempt: 3 I0918 20:39:39.241487 28171 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e: Bootstrap starting. I0918 20:39:39.244864 28171 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e: No blocks or log segments found. Creating new log. I0918 20:39:39.248172 28171 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e: No bootstrap required, opened a new log I0918 20:39:39.248072 28057 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444: Bootstrap starting. I0918 20:39:39.250108 28171 raft_consensus.cc:340] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } I0918 20:39:39.250624 28171 raft_consensus.cc:366] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:39.250831 28171 raft_consensus.cc:705] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80e9151b52aa4ba6bed4e6336b24a35e, State: Initialized, Role: FOLLOWER I0918 20:39:39.251582 28057 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444: No blocks or log segments found. Creating new log. I0918 20:39:39.251369 28171 consensus_queue.cc:229] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } I0918 20:39:39.253223 28171 sys_catalog.cc:340] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } } I0918 20:39:39.253501 28179 sys_catalog.cc:340] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } } I0918 20:39:39.254086 28171 sys_catalog.cc:343] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:39.254364 28179 sys_catalog.cc:343] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:39.255441 28171 sys_catalog.cc:428] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:39.256100 28057 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444: No bootstrap required, opened a new log I0918 20:39:39.257957 28057 raft_consensus.cc:340] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } I0918 20:39:39.258539 28057 raft_consensus.cc:366] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely W0918 20:39:39.258548 28181 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e: acquiring CA information for follower catalog manager: Not found: root CA entry not found I0918 20:39:39.258739 28057 raft_consensus.cc:705] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36211758fed84e26a80a3f1cd3c72444, State: Initialized, Role: FOLLOWER W0918 20:39:39.258886 28181 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:39.259316 28057 consensus_queue.cc:229] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } I0918 20:39:39.260692 28057 sys_catalog.cc:340] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } } I0918 20:39:39.261184 28057 sys_catalog.cc:343] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:39.260877 28182 sys_catalog.cc:340] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } } I0918 20:39:39.261494 28182 sys_catalog.cc:343] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:39.262136 28057 sys_catalog.cc:428] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:39.263101 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 I0918 20:39:39.263285 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 W0918 20:39:39.265002 28183 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:39.265278 28183 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:39.278362 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 167092us I0918 20:39:39.278745 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:39.278921 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:39.279136 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:39.279261 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:39.280329 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-0-root/instance: uuid: "d32ba85bbbe247d7945b09267f723959" format_stamp: "Formatted at 2019-09-18 20:39:39 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:39.290390 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.009s sys 0.001s I0918 20:39:39.292096 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:39.300735 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:39:39.300967 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-0-root uuid: "d32ba85bbbe247d7945b09267f723959" format_stamp: "Formatted at 2019-09-18 20:39:39 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:39.301228 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:39.337954 28175 raft_consensus.cc:472] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:39.338496 28175 raft_consensus.cc:494] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } I0918 20:39:39.339977 28175 leader_election.cc:296] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36211758fed84e26a80a3f1cd3c72444 (127.18.222.62:37137), 80e9151b52aa4ba6bed4e6336b24a35e (127.18.222.60:44875) I0918 20:39:39.340906 28034 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36211758fed84e26a80a3f1cd3c72444" is_pre_election: true I0918 20:39:39.341094 28148 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" is_pre_election: true I0918 20:39:39.341543 28034 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7d0a1df3aa2499bb006d06bfbc57531 in term 0. I0918 20:39:39.341823 28148 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7d0a1df3aa2499bb006d06bfbc57531 in term 0. I0918 20:39:39.342672 28062 leader_election.cc:310] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36211758fed84e26a80a3f1cd3c72444, e7d0a1df3aa2499bb006d06bfbc57531; no voters: I0918 20:39:39.343531 28175 raft_consensus.cc:2644] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:39.343875 28175 raft_consensus.cc:472] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:39.344099 28175 raft_consensus.cc:2890] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:39.348425 28175 raft_consensus.cc:494] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } I0918 20:39:39.349846 28175 leader_election.cc:296] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [CANDIDATE]: Term 1 election: Requested vote from peers 36211758fed84e26a80a3f1cd3c72444 (127.18.222.62:37137), 80e9151b52aa4ba6bed4e6336b24a35e (127.18.222.60:44875) I0918 20:39:39.350793 28034 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36211758fed84e26a80a3f1cd3c72444" I0918 20:39:39.351181 28148 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" I0918 20:39:39.351547 28034 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:39.351894 28148 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:39.356469 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:39.356897 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:39.357442 28034 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7d0a1df3aa2499bb006d06bfbc57531 in term 1. I0918 20:39:39.358876 28062 leader_election.cc:310] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36211758fed84e26a80a3f1cd3c72444, e7d0a1df3aa2499bb006d06bfbc57531; no voters: I0918 20:39:39.359205 28148 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7d0a1df3aa2499bb006d06bfbc57531 in term 1. I0918 20:39:39.359601 28175 raft_consensus.cc:2644] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:39.362299 28175 raft_consensus.cc:667] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [term 1 LEADER]: Becoming Leader. State: Replica: e7d0a1df3aa2499bb006d06bfbc57531, State: Running, Role: LEADER I0918 20:39:39.363070 28175 consensus_queue.cc:206] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } I0918 20:39:39.367277 28195 sys_catalog.cc:340] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7d0a1df3aa2499bb006d06bfbc57531. Latest consensus state: current_term: 1 leader_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } } I0918 20:39:39.368172 28195 sys_catalog.cc:343] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [sys.catalog]: This master's current role is: LEADER I0918 20:39:39.370581 28201 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:39:39.373718 28201 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:39:39.406452 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:38599 I0918 20:39:39.408504 19320 webserver.cc:374] Webserver started at http://127.18.222.1:34181/ using document root and password file I0918 20:39:39.425693 28148 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e [term 1 FOLLOWER]: Refusing update from remote peer e7d0a1df3aa2499bb006d06bfbc57531: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:39.427662 28175 consensus_queue.cc:984] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:39.429141 28034 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444 [term 1 FOLLOWER]: Refusing update from remote peer e7d0a1df3aa2499bb006d06bfbc57531: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:39.430984 28175 consensus_queue.cc:984] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:39.450783 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 167092us I0918 20:39:39.451273 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:39.451505 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:39.451843 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:39.451400 28252 heartbeater.cc:322] Connected to a master server at 127.18.222.61:38767 I0918 20:39:39.452025 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:39.450275 28179 sys_catalog.cc:340] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7d0a1df3aa2499bb006d06bfbc57531. Latest consensus state: current_term: 1 leader_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } } I0918 20:39:39.452365 28252 heartbeater.cc:406] Registering TS with master... I0918 20:39:39.453198 28252 heartbeater.cc:455] Master 127.18.222.61:38767 requested a full tablet report, sending... I0918 20:39:39.452519 28179 sys_catalog.cc:343] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:39.455811 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-1-root/instance: uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" format_stamp: "Formatted at 2019-09-18 20:39:39 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:39.456646 28081 ts_manager.cc:186] Registered new tserver with Master: d32ba85bbbe247d7945b09267f723959 (127.18.222.1:38599) I0918 20:39:39.456249 28182 sys_catalog.cc:340] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7d0a1df3aa2499bb006d06bfbc57531. Latest consensus state: current_term: 1 leader_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } } I0918 20:39:39.457888 28182 sys_catalog.cc:343] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:39.460170 28251 heartbeater.cc:322] Connected to a master server at 127.18.222.60:44875 I0918 20:39:39.460642 28251 heartbeater.cc:406] Registering TS with master... I0918 20:39:39.461477 28251 heartbeater.cc:455] Master 127.18.222.60:44875 requested a full tablet report, sending... I0918 20:39:39.464169 28253 heartbeater.cc:322] Connected to a master server at 127.18.222.62:37137 I0918 20:39:39.464372 28138 ts_manager.cc:186] Registered new tserver with Master: d32ba85bbbe247d7945b09267f723959 (127.18.222.1:38599) I0918 20:39:39.464756 28253 heartbeater.cc:406] Registering TS with master... I0918 20:39:39.465430 28253 heartbeater.cc:455] Master 127.18.222.62:37137 requested a full tablet report, sending... I0918 20:39:39.468372 28024 ts_manager.cc:186] Registered new tserver with Master: d32ba85bbbe247d7945b09267f723959 (127.18.222.1:38599) I0918 20:39:39.472270 28175 sys_catalog.cc:340] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } } I0918 20:39:39.473073 28175 sys_catalog.cc:343] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [sys.catalog]: This master's current role is: LEADER I0918 20:39:39.473651 28175 sys_catalog.cc:340] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } } I0918 20:39:39.474318 28175 sys_catalog.cc:343] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [sys.catalog]: This master's current role is: LEADER I0918 20:39:39.478710 28182 sys_catalog.cc:340] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } } I0918 20:39:39.479789 28182 sys_catalog.cc:343] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:39.482148 28201 catalog_manager.cc:981] Generated new certificate authority record I0918 20:39:39.485440 28201 catalog_manager.cc:1106] Loading token signing keys... I0918 20:39:39.488164 28179 sys_catalog.cc:340] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36211758fed84e26a80a3f1cd3c72444" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 37137 } } peers { permanent_uuid: "e7d0a1df3aa2499bb006d06bfbc57531" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 38767 } } peers { permanent_uuid: "80e9151b52aa4ba6bed4e6336b24a35e" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 44875 } } } I0918 20:39:39.489078 28179 sys_catalog.cc:343] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:39.492668 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.036s user 0.015s sys 0.007s I0918 20:39:39.494904 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:39.510080 28201 catalog_manager.cc:4446] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531: Generated new TSK 0 I0918 20:39:39.511106 28201 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:39:39.516391 19320 fs_manager.cc:419] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s I0918 20:39:39.516790 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-1-root uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" format_stamp: "Formatted at 2019-09-18 20:39:39 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:39.517134 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:39.517953 28081 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34456 I0918 20:39:39.532141 28252 heartbeater.cc:447] Master 127.18.222.61:38767 was elected leader, sending a full tablet report... I0918 20:39:39.560122 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:39.560411 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:39.619668 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.2:46789 I0918 20:39:39.621568 19320 webserver.cc:374] Webserver started at http://127.18.222.2:41187/ using document root and password file I0918 20:39:39.670495 28332 heartbeater.cc:322] Connected to a master server at 127.18.222.61:38767 I0918 20:39:39.672410 28332 heartbeater.cc:406] Registering TS with master... I0918 20:39:39.672899 28332 heartbeater.cc:455] Master 127.18.222.61:38767 requested a full tablet report, sending... I0918 20:39:39.673029 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 167092us I0918 20:39:39.673597 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:39.673871 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:39.673766 28331 heartbeater.cc:322] Connected to a master server at 127.18.222.60:44875 I0918 20:39:39.674335 28331 heartbeater.cc:406] Registering TS with master... I0918 20:39:39.674762 28331 heartbeater.cc:455] Master 127.18.222.60:44875 requested a full tablet report, sending... I0918 20:39:39.674767 28081 ts_manager.cc:186] Registered new tserver with Master: a68e2c30ad9f4733bbf8eb54594ce86c (127.18.222.2:46789) I0918 20:39:39.675489 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:39.675778 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:39.676746 28081 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34462 I0918 20:39:39.677137 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-2-root/instance: uuid: "8654236fd4b14655929a78c02e6e8021" format_stamp: "Formatted at 2019-09-18 20:39:39 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:39.677229 28138 ts_manager.cc:186] Registered new tserver with Master: a68e2c30ad9f4733bbf8eb54594ce86c (127.18.222.2:46789) I0918 20:39:39.680961 28333 heartbeater.cc:322] Connected to a master server at 127.18.222.62:37137 I0918 20:39:39.681252 28333 heartbeater.cc:406] Registering TS with master... I0918 20:39:39.681737 28333 heartbeater.cc:455] Master 127.18.222.62:37137 requested a full tablet report, sending... I0918 20:39:39.684164 28024 ts_manager.cc:186] Registered new tserver with Master: a68e2c30ad9f4733bbf8eb54594ce86c (127.18.222.2:46789) I0918 20:39:39.690568 28332 heartbeater.cc:447] Master 127.18.222.61:38767 was elected leader, sending a full tablet report... I0918 20:39:39.692442 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.014s user 0.013s sys 0.000s I0918 20:39:39.694396 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:39.707273 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:39.707509 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-2-root uuid: "8654236fd4b14655929a78c02e6e8021" format_stamp: "Formatted at 2019-09-18 20:39:39 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:39.707870 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1568839089958813-19320/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:39.781780 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:39.782124 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:39.843947 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.3:41335 I0918 20:39:39.846645 19320 webserver.cc:374] Webserver started at http://127.18.222.3:36409/ using document root and password file I0918 20:39:39.875125 28402 heartbeater.cc:322] Connected to a master server at 127.18.222.60:44875 I0918 20:39:39.875659 28402 heartbeater.cc:406] Registering TS with master... I0918 20:39:39.876574 28402 heartbeater.cc:455] Master 127.18.222.60:44875 requested a full tablet report, sending... I0918 20:39:39.879204 28138 ts_manager.cc:186] Registered new tserver with Master: 8654236fd4b14655929a78c02e6e8021 (127.18.222.3:41335) I0918 20:39:39.884997 28403 heartbeater.cc:322] Connected to a master server at 127.18.222.61:38767 I0918 20:39:39.885511 28403 heartbeater.cc:406] Registering TS with master... I0918 20:39:39.886087 28403 heartbeater.cc:455] Master 127.18.222.61:38767 requested a full tablet report, sending... I0918 20:39:39.888181 28080 ts_manager.cc:186] Registered new tserver with Master: 8654236fd4b14655929a78c02e6e8021 (127.18.222.3:41335) I0918 20:39:39.890336 28080 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34468 I0918 20:39:39.897352 28405 heartbeater.cc:322] Connected to a master server at 127.18.222.62:37137 I0918 20:39:39.897681 28405 heartbeater.cc:406] Registering TS with master... I0918 20:39:39.898205 28405 heartbeater.cc:455] Master 127.18.222.62:37137 requested a full tablet report, sending... I0918 20:39:39.900591 28023 ts_manager.cc:186] Registered new tserver with Master: 8654236fd4b14655929a78c02e6e8021 (127.18.222.3:41335) I0918 20:39:39.902240 19320 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.048350078s I0918 20:39:39.904935 28403 heartbeater.cc:447] Master 127.18.222.61:38767 was elected leader, sending a full tablet report... I0918 20:39:39.986937 28080 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34476: name: "ksck-test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\n\000\000\000\004\001\024\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:39:39.988502 28080 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:39:40.137723 28298 tablet_service.cc:1022] Processing CreateTablet for tablet 0e7777829e0c46a3a5dc016b95c26a5b (table=ksck-test-table [id=a51f44f49d18445d9cdaaafed6d128b6]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:40.138804 28298 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 0e7777829e0c46a3a5dc016b95c26a5b. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:40.144803 28297 tablet_service.cc:1022] Processing CreateTablet for tablet 20884cc5471a4921b39674fa4462064c (table=ksck-test-table [id=a51f44f49d18445d9cdaaafed6d128b6]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:40.146061 28297 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 20884cc5471a4921b39674fa4462064c. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:40.147161 28218 tablet_service.cc:1022] Processing CreateTablet for tablet 0e7777829e0c46a3a5dc016b95c26a5b (table=ksck-test-table [id=a51f44f49d18445d9cdaaafed6d128b6]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:40.152947 28296 tablet_service.cc:1022] Processing CreateTablet for tablet 7198354ae5254f8c92805487d4542a83 (table=ksck-test-table [id=a51f44f49d18445d9cdaaafed6d128b6]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:40.154105 28296 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 7198354ae5254f8c92805487d4542a83. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:40.154830 28218 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 0e7777829e0c46a3a5dc016b95c26a5b. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:40.161031 28217 tablet_service.cc:1022] Processing CreateTablet for tablet 20884cc5471a4921b39674fa4462064c (table=ksck-test-table [id=a51f44f49d18445d9cdaaafed6d128b6]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:40.161317 28216 tablet_service.cc:1022] Processing CreateTablet for tablet 7198354ae5254f8c92805487d4542a83 (table=ksck-test-table [id=a51f44f49d18445d9cdaaafed6d128b6]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:40.162322 28217 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 20884cc5471a4921b39674fa4462064c. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:40.166165 28216 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 7198354ae5254f8c92805487d4542a83. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:40.182415 28369 tablet_service.cc:1022] Processing CreateTablet for tablet 0e7777829e0c46a3a5dc016b95c26a5b (table=ksck-test-table [id=a51f44f49d18445d9cdaaafed6d128b6]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:40.183622 28369 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 0e7777829e0c46a3a5dc016b95c26a5b. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:40.186761 28368 tablet_service.cc:1022] Processing CreateTablet for tablet 20884cc5471a4921b39674fa4462064c (table=ksck-test-table [id=a51f44f49d18445d9cdaaafed6d128b6]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:40.188647 28367 tablet_service.cc:1022] Processing CreateTablet for tablet 7198354ae5254f8c92805487d4542a83 (table=ksck-test-table [id=a51f44f49d18445d9cdaaafed6d128b6]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:40.195212 28367 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 7198354ae5254f8c92805487d4542a83. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:40.199379 28368 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 20884cc5471a4921b39674fa4462064c. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:40.275362 28424 tablet_bootstrap.cc:439] T 7198354ae5254f8c92805487d4542a83 P d32ba85bbbe247d7945b09267f723959: Bootstrap starting. I0918 20:39:40.273121 28423 tablet_bootstrap.cc:439] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c: Bootstrap starting. I0918 20:39:40.299160 28181 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e: acquiring CA information for follower catalog manager: success I0918 20:39:40.300241 28183 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444: acquiring CA information for follower catalog manager: success I0918 20:39:40.336264 28183 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:40.337535 28181 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:40.343456 28425 tablet_bootstrap.cc:439] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021: Bootstrap starting. I0918 20:39:40.346390 28425 tablet_bootstrap.cc:590] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021: No blocks or log segments found. Creating new log. I0918 20:39:40.342154 28423 tablet_bootstrap.cc:590] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c: No blocks or log segments found. Creating new log. I0918 20:39:40.348870 28424 tablet_bootstrap.cc:590] T 7198354ae5254f8c92805487d4542a83 P d32ba85bbbe247d7945b09267f723959: No blocks or log segments found. Creating new log. I0918 20:39:40.359874 28424 tablet_bootstrap.cc:439] T 7198354ae5254f8c92805487d4542a83 P d32ba85bbbe247d7945b09267f723959: No bootstrap required, opened a new log I0918 20:39:40.364189 28423 tablet_bootstrap.cc:439] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c: No bootstrap required, opened a new log I0918 20:39:40.368592 28424 ts_tablet_manager.cc:1085] T 7198354ae5254f8c92805487d4542a83 P d32ba85bbbe247d7945b09267f723959: Time spent bootstrapping tablet: real 0.093s user 0.013s sys 0.000s I0918 20:39:40.368739 28423 ts_tablet_manager.cc:1085] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c: Time spent bootstrapping tablet: real 0.096s user 0.008s sys 0.001s I0918 20:39:40.372612 28425 tablet_bootstrap.cc:439] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021: No bootstrap required, opened a new log I0918 20:39:40.373272 28425 ts_tablet_manager.cc:1085] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.008s I0918 20:39:40.375728 28425 raft_consensus.cc:340] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } I0918 20:39:40.376586 28425 raft_consensus.cc:366] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:40.376864 28425 raft_consensus.cc:705] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8654236fd4b14655929a78c02e6e8021, State: Initialized, Role: FOLLOWER I0918 20:39:40.377661 28425 consensus_queue.cc:229] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } I0918 20:39:40.382210 28424 raft_consensus.cc:340] T 7198354ae5254f8c92805487d4542a83 P d32ba85bbbe247d7945b09267f723959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } I0918 20:39:40.383272 28424 raft_consensus.cc:366] T 7198354ae5254f8c92805487d4542a83 P d32ba85bbbe247d7945b09267f723959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:40.377307 28423 raft_consensus.cc:340] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } I0918 20:39:40.383949 28423 raft_consensus.cc:366] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:40.384251 28423 raft_consensus.cc:705] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68e2c30ad9f4733bbf8eb54594ce86c, State: Initialized, Role: FOLLOWER I0918 20:39:40.383558 28424 raft_consensus.cc:705] T 7198354ae5254f8c92805487d4542a83 P d32ba85bbbe247d7945b09267f723959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d32ba85bbbe247d7945b09267f723959, State: Initialized, Role: FOLLOWER I0918 20:39:40.386914 28424 consensus_queue.cc:229] T 7198354ae5254f8c92805487d4542a83 P d32ba85bbbe247d7945b09267f723959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } I0918 20:39:40.385835 28423 consensus_queue.cc:229] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } I0918 20:39:40.397219 28425 ts_tablet_manager.cc:1113] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021: Time spent starting tablet: real 0.024s user 0.006s sys 0.004s I0918 20:39:40.401049 28425 tablet_bootstrap.cc:439] T 20884cc5471a4921b39674fa4462064c P 8654236fd4b14655929a78c02e6e8021: Bootstrap starting. I0918 20:39:40.407050 28424 ts_tablet_manager.cc:1113] T 7198354ae5254f8c92805487d4542a83 P d32ba85bbbe247d7945b09267f723959: Time spent starting tablet: real 0.037s user 0.008s sys 0.000s I0918 20:39:40.413403 28425 tablet_bootstrap.cc:590] T 20884cc5471a4921b39674fa4462064c P 8654236fd4b14655929a78c02e6e8021: No blocks or log segments found. Creating new log. I0918 20:39:40.417542 28423 ts_tablet_manager.cc:1113] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c: Time spent starting tablet: real 0.046s user 0.006s sys 0.001s I0918 20:39:40.418484 28423 tablet_bootstrap.cc:439] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c: Bootstrap starting. I0918 20:39:40.422118 28423 tablet_bootstrap.cc:590] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c: No blocks or log segments found. Creating new log. I0918 20:39:40.424352 28424 tablet_bootstrap.cc:439] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959: Bootstrap starting. I0918 20:39:40.426920 28423 tablet_bootstrap.cc:439] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c: No bootstrap required, opened a new log I0918 20:39:40.431608 28423 ts_tablet_manager.cc:1085] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s I0918 20:39:40.433951 28423 raft_consensus.cc:340] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } I0918 20:39:40.434494 28423 raft_consensus.cc:366] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:40.434708 28423 raft_consensus.cc:705] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68e2c30ad9f4733bbf8eb54594ce86c, State: Initialized, Role: FOLLOWER I0918 20:39:40.438972 28424 tablet_bootstrap.cc:590] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959: No blocks or log segments found. Creating new log. I0918 20:39:40.435298 28423 consensus_queue.cc:229] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } I0918 20:39:40.441489 28425 tablet_bootstrap.cc:439] T 20884cc5471a4921b39674fa4462064c P 8654236fd4b14655929a78c02e6e8021: No bootstrap required, opened a new log I0918 20:39:40.442096 28425 ts_tablet_manager.cc:1085] T 20884cc5471a4921b39674fa4462064c P 8654236fd4b14655929a78c02e6e8021: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.005s I0918 20:39:40.444412 28425 raft_consensus.cc:340] T 20884cc5471a4921b39674fa4462064c P 8654236fd4b14655929a78c02e6e8021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } I0918 20:39:40.467945 28425 raft_consensus.cc:366] T 20884cc5471a4921b39674fa4462064c P 8654236fd4b14655929a78c02e6e8021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:40.447227 28423 ts_tablet_manager.cc:1113] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c: Time spent starting tablet: real 0.015s user 0.002s sys 0.002s I0918 20:39:40.469038 28425 raft_consensus.cc:705] T 20884cc5471a4921b39674fa4462064c P 8654236fd4b14655929a78c02e6e8021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8654236fd4b14655929a78c02e6e8021, State: Initialized, Role: FOLLOWER I0918 20:39:40.469977 28424 tablet_bootstrap.cc:439] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959: No bootstrap required, opened a new log I0918 20:39:40.470660 28423 tablet_bootstrap.cc:439] T 20884cc5471a4921b39674fa4462064c P a68e2c30ad9f4733bbf8eb54594ce86c: Bootstrap starting. I0918 20:39:40.470088 28425 consensus_queue.cc:229] T 20884cc5471a4921b39674fa4462064c P 8654236fd4b14655929a78c02e6e8021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } I0918 20:39:40.470798 28424 ts_tablet_manager.cc:1085] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959: Time spent bootstrapping tablet: real 0.047s user 0.021s sys 0.008s I0918 20:39:40.472297 28425 ts_tablet_manager.cc:1113] T 20884cc5471a4921b39674fa4462064c P 8654236fd4b14655929a78c02e6e8021: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s I0918 20:39:40.473376 28425 tablet_bootstrap.cc:439] T 7198354ae5254f8c92805487d4542a83 P 8654236fd4b14655929a78c02e6e8021: Bootstrap starting. I0918 20:39:40.475021 28424 raft_consensus.cc:340] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } I0918 20:39:40.478708 28431 raft_consensus.cc:472] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:40.479840 28431 raft_consensus.cc:494] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } I0918 20:39:40.484014 28431 leader_election.cc:296] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8654236fd4b14655929a78c02e6e8021 (127.18.222.3:41335), d32ba85bbbe247d7945b09267f723959 (127.18.222.1:38599) I0918 20:39:40.478519 28424 raft_consensus.cc:366] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:40.489823 28424 raft_consensus.cc:705] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d32ba85bbbe247d7945b09267f723959, State: Initialized, Role: FOLLOWER I0918 20:39:40.490479 28424 consensus_queue.cc:229] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } I0918 20:39:40.500743 28424 ts_tablet_manager.cc:1113] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959: Time spent starting tablet: real 0.030s user 0.005s sys 0.001s I0918 20:39:40.507436 28424 tablet_bootstrap.cc:439] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959: Bootstrap starting. I0918 20:39:40.513108 28423 tablet_bootstrap.cc:590] T 20884cc5471a4921b39674fa4462064c P a68e2c30ad9f4733bbf8eb54594ce86c: No blocks or log segments found. Creating new log. I0918 20:39:40.508993 28425 tablet_bootstrap.cc:590] T 7198354ae5254f8c92805487d4542a83 P 8654236fd4b14655929a78c02e6e8021: No blocks or log segments found. Creating new log. I0918 20:39:40.551038 28423 tablet_bootstrap.cc:439] T 20884cc5471a4921b39674fa4462064c P a68e2c30ad9f4733bbf8eb54594ce86c: No bootstrap required, opened a new log I0918 20:39:40.553504 28425 tablet_bootstrap.cc:439] T 7198354ae5254f8c92805487d4542a83 P 8654236fd4b14655929a78c02e6e8021: No bootstrap required, opened a new log I0918 20:39:40.552237 28423 ts_tablet_manager.cc:1085] T 20884cc5471a4921b39674fa4462064c P a68e2c30ad9f4733bbf8eb54594ce86c: Time spent bootstrapping tablet: real 0.082s user 0.007s sys 0.005s I0918 20:39:40.555297 28425 ts_tablet_manager.cc:1085] T 7198354ae5254f8c92805487d4542a83 P 8654236fd4b14655929a78c02e6e8021: Time spent bootstrapping tablet: real 0.082s user 0.013s sys 0.002s I0918 20:39:40.555974 28424 tablet_bootstrap.cc:590] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959: No blocks or log segments found. Creating new log. I0918 20:39:40.558097 28425 raft_consensus.cc:340] T 7198354ae5254f8c92805487d4542a83 P 8654236fd4b14655929a78c02e6e8021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } I0918 20:39:40.558969 28425 raft_consensus.cc:366] T 7198354ae5254f8c92805487d4542a83 P 8654236fd4b14655929a78c02e6e8021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:40.559255 28425 raft_consensus.cc:705] T 7198354ae5254f8c92805487d4542a83 P 8654236fd4b14655929a78c02e6e8021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8654236fd4b14655929a78c02e6e8021, State: Initialized, Role: FOLLOWER I0918 20:39:40.560134 28425 consensus_queue.cc:229] T 7198354ae5254f8c92805487d4542a83 P 8654236fd4b14655929a78c02e6e8021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } I0918 20:39:40.550742 28379 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0e7777829e0c46a3a5dc016b95c26a5b" candidate_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8654236fd4b14655929a78c02e6e8021" is_pre_election: true I0918 20:39:40.561501 28423 raft_consensus.cc:340] T 20884cc5471a4921b39674fa4462064c P a68e2c30ad9f4733bbf8eb54594ce86c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } I0918 20:39:40.562481 28423 raft_consensus.cc:366] T 20884cc5471a4921b39674fa4462064c P a68e2c30ad9f4733bbf8eb54594ce86c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:40.562530 28379 raft_consensus.cc:2350] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a68e2c30ad9f4733bbf8eb54594ce86c in term 0. I0918 20:39:40.562819 28423 raft_consensus.cc:705] T 20884cc5471a4921b39674fa4462064c P a68e2c30ad9f4733bbf8eb54594ce86c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68e2c30ad9f4733bbf8eb54594ce86c, State: Initialized, Role: FOLLOWER I0918 20:39:40.564014 28271 leader_election.cc:310] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8654236fd4b14655929a78c02e6e8021, a68e2c30ad9f4733bbf8eb54594ce86c; no voters: I0918 20:39:40.563746 28423 consensus_queue.cc:229] T 20884cc5471a4921b39674fa4462064c P a68e2c30ad9f4733bbf8eb54594ce86c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } I0918 20:39:40.565057 28431 raft_consensus.cc:2644] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:40.565845 28431 raft_consensus.cc:472] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:40.566128 28431 raft_consensus.cc:2890] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:40.576459 28423 ts_tablet_manager.cc:1113] T 20884cc5471a4921b39674fa4462064c P a68e2c30ad9f4733bbf8eb54594ce86c: Time spent starting tablet: real 0.021s user 0.007s sys 0.004s I0918 20:39:40.578316 28431 raft_consensus.cc:494] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } I0918 20:39:40.581542 28379 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0e7777829e0c46a3a5dc016b95c26a5b" candidate_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8654236fd4b14655929a78c02e6e8021" I0918 20:39:40.582180 28379 raft_consensus.cc:2890] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:40.589772 28379 raft_consensus.cc:2350] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a68e2c30ad9f4733bbf8eb54594ce86c in term 1. I0918 20:39:40.591074 28271 leader_election.cc:310] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8654236fd4b14655929a78c02e6e8021, a68e2c30ad9f4733bbf8eb54594ce86c; no voters: I0918 20:39:40.590881 28228 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0e7777829e0c46a3a5dc016b95c26a5b" candidate_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d32ba85bbbe247d7945b09267f723959" is_pre_election: true W0918 20:39:40.593044 28270 leader_election.cc:348] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d32ba85bbbe247d7945b09267f723959 (127.18.222.1:38599): Illegal state: must be running to vote when last-logged opid is not known I0918 20:39:40.594408 28440 raft_consensus.cc:2644] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:40.595803 28424 tablet_bootstrap.cc:439] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959: No bootstrap required, opened a new log I0918 20:39:40.596518 28424 ts_tablet_manager.cc:1085] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959: Time spent bootstrapping tablet: real 0.089s user 0.006s sys 0.023s I0918 20:39:40.596609 28228 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0e7777829e0c46a3a5dc016b95c26a5b" candidate_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d32ba85bbbe247d7945b09267f723959" I0918 20:39:40.597368 28431 leader_election.cc:296] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [CANDIDATE]: Term 1 election: Requested vote from peers 8654236fd4b14655929a78c02e6e8021 (127.18.222.3:41335), d32ba85bbbe247d7945b09267f723959 (127.18.222.1:38599) I0918 20:39:40.599557 28424 raft_consensus.cc:340] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } I0918 20:39:40.600397 28424 raft_consensus.cc:366] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:40.600678 28424 raft_consensus.cc:705] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d32ba85bbbe247d7945b09267f723959, State: Initialized, Role: FOLLOWER I0918 20:39:40.601348 28424 consensus_queue.cc:229] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } W0918 20:39:40.601824 28270 leader_election.cc:348] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d32ba85bbbe247d7945b09267f723959 (127.18.222.1:38599): Illegal state: must be running to vote when last-logged opid is not known I0918 20:39:40.604321 28425 ts_tablet_manager.cc:1113] T 7198354ae5254f8c92805487d4542a83 P 8654236fd4b14655929a78c02e6e8021: Time spent starting tablet: real 0.049s user 0.005s sys 0.000s I0918 20:39:40.607178 28440 raft_consensus.cc:667] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 1 LEADER]: Becoming Leader. State: Replica: a68e2c30ad9f4733bbf8eb54594ce86c, State: Running, Role: LEADER I0918 20:39:40.608036 28440 consensus_queue.cc:206] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } I0918 20:39:40.613386 28430 raft_consensus.cc:472] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:40.614089 28430 raft_consensus.cc:494] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } I0918 20:39:40.609354 28424 ts_tablet_manager.cc:1113] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s I0918 20:39:40.617931 28430 leader_election.cc:296] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a68e2c30ad9f4733bbf8eb54594ce86c (127.18.222.2:46789), 8654236fd4b14655929a78c02e6e8021 (127.18.222.3:41335) I0918 20:39:40.623934 28441 raft_consensus.cc:472] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:40.623520 28080 catalog_manager.cc:4131] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c reported cstate change: term changed from 0 to 1, leader changed from to a68e2c30ad9f4733bbf8eb54594ce86c (127.18.222.2). New cstate: current_term: 1 leader_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } health_report { overall_health: UNKNOWN } } } I0918 20:39:40.628021 28441 raft_consensus.cc:494] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } I0918 20:39:40.645020 28228 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7198354ae5254f8c92805487d4542a83" candidate_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d32ba85bbbe247d7945b09267f723959" is_pre_election: true I0918 20:39:40.645823 28228 raft_consensus.cc:2350] T 7198354ae5254f8c92805487d4542a83 P d32ba85bbbe247d7945b09267f723959 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a68e2c30ad9f4733bbf8eb54594ce86c in term 0. I0918 20:39:40.647119 28270 leader_election.cc:310] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a68e2c30ad9f4733bbf8eb54594ce86c, d32ba85bbbe247d7945b09267f723959; no voters: I0918 20:39:40.648144 28440 raft_consensus.cc:2644] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:40.648468 28440 raft_consensus.cc:472] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:40.648749 28440 raft_consensus.cc:2890] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:40.661682 28440 raft_consensus.cc:494] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } I0918 20:39:40.664607 28228 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7198354ae5254f8c92805487d4542a83" candidate_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d32ba85bbbe247d7945b09267f723959" I0918 20:39:40.665263 28228 raft_consensus.cc:2890] T 7198354ae5254f8c92805487d4542a83 P d32ba85bbbe247d7945b09267f723959 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:40.666060 28379 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7198354ae5254f8c92805487d4542a83" candidate_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8654236fd4b14655929a78c02e6e8021" is_pre_election: true I0918 20:39:40.666816 28379 raft_consensus.cc:2350] T 7198354ae5254f8c92805487d4542a83 P 8654236fd4b14655929a78c02e6e8021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a68e2c30ad9f4733bbf8eb54594ce86c in term 0. I0918 20:39:40.667799 28441 leader_election.cc:296] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d32ba85bbbe247d7945b09267f723959 (127.18.222.1:38599), 8654236fd4b14655929a78c02e6e8021 (127.18.222.3:41335) I0918 20:39:40.670294 28379 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7198354ae5254f8c92805487d4542a83" candidate_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8654236fd4b14655929a78c02e6e8021" I0918 20:39:40.670876 28379 raft_consensus.cc:2890] T 7198354ae5254f8c92805487d4542a83 P 8654236fd4b14655929a78c02e6e8021 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:40.677279 28228 raft_consensus.cc:2350] T 7198354ae5254f8c92805487d4542a83 P d32ba85bbbe247d7945b09267f723959 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a68e2c30ad9f4733bbf8eb54594ce86c in term 1. I0918 20:39:40.678766 28270 leader_election.cc:310] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a68e2c30ad9f4733bbf8eb54594ce86c, d32ba85bbbe247d7945b09267f723959; no voters: I0918 20:39:40.679841 28431 raft_consensus.cc:2644] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:40.669206 28440 leader_election.cc:296] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c [CANDIDATE]: Term 1 election: Requested vote from peers d32ba85bbbe247d7945b09267f723959 (127.18.222.1:38599), 8654236fd4b14655929a78c02e6e8021 (127.18.222.3:41335) I0918 20:39:40.698936 28378 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "20884cc5471a4921b39674fa4462064c" candidate_uuid: "d32ba85bbbe247d7945b09267f723959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8654236fd4b14655929a78c02e6e8021" is_pre_election: true I0918 20:39:40.702353 28378 raft_consensus.cc:2350] T 20884cc5471a4921b39674fa4462064c P 8654236fd4b14655929a78c02e6e8021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d32ba85bbbe247d7945b09267f723959 in term 0. I0918 20:39:40.704079 28188 leader_election.cc:310] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8654236fd4b14655929a78c02e6e8021, d32ba85bbbe247d7945b09267f723959; no voters: I0918 20:39:40.705150 28430 raft_consensus.cc:2644] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:40.707908 28430 raft_consensus.cc:472] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:40.708128 28308 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "20884cc5471a4921b39674fa4462064c" candidate_uuid: "d32ba85bbbe247d7945b09267f723959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" is_pre_election: true I0918 20:39:40.708968 28308 raft_consensus.cc:2350] T 20884cc5471a4921b39674fa4462064c P a68e2c30ad9f4733bbf8eb54594ce86c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d32ba85bbbe247d7945b09267f723959 in term 0. I0918 20:39:40.708262 28430 raft_consensus.cc:2890] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:40.703588 28379 raft_consensus.cc:2350] T 7198354ae5254f8c92805487d4542a83 P 8654236fd4b14655929a78c02e6e8021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a68e2c30ad9f4733bbf8eb54594ce86c in term 1. I0918 20:39:40.718657 28431 raft_consensus.cc:667] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c [term 1 LEADER]: Becoming Leader. State: Replica: a68e2c30ad9f4733bbf8eb54594ce86c, State: Running, Role: LEADER I0918 20:39:40.719933 28431 consensus_queue.cc:206] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } I0918 20:39:40.721220 28430 raft_consensus.cc:494] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } I0918 20:39:40.734671 28308 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "20884cc5471a4921b39674fa4462064c" candidate_uuid: "d32ba85bbbe247d7945b09267f723959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" I0918 20:39:40.735594 28308 raft_consensus.cc:2890] T 20884cc5471a4921b39674fa4462064c P a68e2c30ad9f4733bbf8eb54594ce86c [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:40.745000 28379 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "20884cc5471a4921b39674fa4462064c" candidate_uuid: "d32ba85bbbe247d7945b09267f723959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8654236fd4b14655929a78c02e6e8021" I0918 20:39:40.745847 28379 raft_consensus.cc:2890] T 20884cc5471a4921b39674fa4462064c P 8654236fd4b14655929a78c02e6e8021 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:40.733503 28430 leader_election.cc:296] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 [CANDIDATE]: Term 1 election: Requested vote from peers a68e2c30ad9f4733bbf8eb54594ce86c (127.18.222.2:46789), 8654236fd4b14655929a78c02e6e8021 (127.18.222.3:41335) I0918 20:39:40.758484 28379 raft_consensus.cc:2350] T 20884cc5471a4921b39674fa4462064c P 8654236fd4b14655929a78c02e6e8021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d32ba85bbbe247d7945b09267f723959 in term 1. I0918 20:39:40.760069 28188 leader_election.cc:310] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8654236fd4b14655929a78c02e6e8021, d32ba85bbbe247d7945b09267f723959; no voters: I0918 20:39:40.761672 28430 raft_consensus.cc:2644] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:40.760046 28308 raft_consensus.cc:2350] T 20884cc5471a4921b39674fa4462064c P a68e2c30ad9f4733bbf8eb54594ce86c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d32ba85bbbe247d7945b09267f723959 in term 1. W0918 20:39:40.776321 28062 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.222.62:37137, user_credentials={real_user=slave}} blocked reactor thread for 43681.3us I0918 20:39:40.788674 28430 raft_consensus.cc:667] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 [term 1 LEADER]: Becoming Leader. State: Replica: d32ba85bbbe247d7945b09267f723959, State: Running, Role: LEADER I0918 20:39:40.790477 28430 consensus_queue.cc:206] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } I0918 20:39:40.806232 28079 catalog_manager.cc:4131] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c reported cstate change: term changed from 0 to 1, leader changed from to a68e2c30ad9f4733bbf8eb54594ce86c (127.18.222.2). New cstate: current_term: 1 leader_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } health_report { overall_health: UNKNOWN } } } I0918 20:39:40.805980 28080 catalog_manager.cc:4131] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 reported cstate change: term changed from 0 to 1, leader changed from to d32ba85bbbe247d7945b09267f723959 (127.18.222.1). New cstate: current_term: 1 leader_uuid: "d32ba85bbbe247d7945b09267f723959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } health_report { overall_health: UNKNOWN } } } I0918 20:39:40.955962 19320 ksck_remote-test.cc:749] Generating row writes... I0918 20:39:41.084180 28228 raft_consensus.cc:2890] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:41.107385 28430 raft_consensus.cc:472] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:41.110841 28430 raft_consensus.cc:494] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } I0918 20:39:41.115631 28379 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0e7777829e0c46a3a5dc016b95c26a5b" candidate_uuid: "d32ba85bbbe247d7945b09267f723959" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8654236fd4b14655929a78c02e6e8021" is_pre_election: true I0918 20:39:41.116341 28308 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0e7777829e0c46a3a5dc016b95c26a5b" candidate_uuid: "d32ba85bbbe247d7945b09267f723959" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" is_pre_election: true I0918 20:39:41.117142 28379 raft_consensus.cc:2350] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d32ba85bbbe247d7945b09267f723959 in term 1. I0918 20:39:41.117591 28308 raft_consensus.cc:2311] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate d32ba85bbbe247d7945b09267f723959 for term 2 because replica is either leader or believes a valid leader to be alive. I0918 20:39:41.118750 28188 leader_election.cc:310] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8654236fd4b14655929a78c02e6e8021, d32ba85bbbe247d7945b09267f723959; no voters: I0918 20:39:41.116503 28430 leader_election.cc:296] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a68e2c30ad9f4733bbf8eb54594ce86c (127.18.222.2:46789), 8654236fd4b14655929a78c02e6e8021 (127.18.222.3:41335) I0918 20:39:41.122618 28431 consensus_queue.cc:984] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:39:41.132249 28430 raft_consensus.cc:2644] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0918 20:39:41.133160 28430 raft_consensus.cc:472] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a68e2c30ad9f4733bbf8eb54594ce86c) I0918 20:39:41.133560 28430 raft_consensus.cc:2890] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 1 FOLLOWER]: Advancing to term 2 I0918 20:39:41.138862 28431 consensus_queue.cc:984] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:41.143270 28430 raft_consensus.cc:494] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } I0918 20:39:41.172982 28308 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0e7777829e0c46a3a5dc016b95c26a5b" candidate_uuid: "d32ba85bbbe247d7945b09267f723959" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" I0918 20:39:41.187243 28430 leader_election.cc:296] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [CANDIDATE]: Term 2 election: Requested vote from peers a68e2c30ad9f4733bbf8eb54594ce86c (127.18.222.2:46789), 8654236fd4b14655929a78c02e6e8021 (127.18.222.3:41335) I0918 20:39:41.192515 28378 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0e7777829e0c46a3a5dc016b95c26a5b" candidate_uuid: "d32ba85bbbe247d7945b09267f723959" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8654236fd4b14655929a78c02e6e8021" I0918 20:39:41.207235 28308 raft_consensus.cc:2311] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 1 LEADER]: Leader election vote request: Denying vote to candidate d32ba85bbbe247d7945b09267f723959 for term 2 because replica is either leader or believes a valid leader to be alive. I0918 20:39:41.210484 28430 consensus_queue.cc:984] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:41.243902 28431 consensus_queue.cc:984] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:41.276161 28228 raft_consensus.cc:1149] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 2 FOLLOWER]: Rejecting Update request from peer a68e2c30ad9f4733bbf8eb54594ce86c for earlier term 1. Current term is 2. Ops: [] I0918 20:39:41.289120 28447 consensus_queue.cc:995] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.006s I0918 20:39:41.296561 28431 raft_consensus.cc:2885] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 1 LEADER]: Stepping down as leader of term 1 I0918 20:39:41.296885 28431 raft_consensus.cc:705] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a68e2c30ad9f4733bbf8eb54594ce86c, State: Running, Role: LEADER I0918 20:39:41.297727 28431 consensus_queue.cc:229] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } I0918 20:39:41.298574 28441 consensus_queue.cc:984] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:39:41.298738 28431 raft_consensus.cc:2890] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 1 FOLLOWER]: Advancing to term 2 I0918 20:39:41.352223 28378 raft_consensus.cc:2325] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d32ba85bbbe247d7945b09267f723959 for term 2 because replica is already servicing an update from a current leader or another vote. W0918 20:39:41.374421 28186 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.222.2:46789, user_credentials={real_user=slave}} blocked reactor thread for 50515.7us I0918 20:39:41.378800 28430 consensus_queue.cc:984] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s W0918 20:39:41.382254 28287 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:51164: Illegal state: Replica a68e2c30ad9f4733bbf8eb54594ce86c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } I0918 20:39:41.385190 28188 leader_election.cc:310] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d32ba85bbbe247d7945b09267f723959; no voters: 8654236fd4b14655929a78c02e6e8021, a68e2c30ad9f4733bbf8eb54594ce86c I0918 20:39:41.402264 28430 raft_consensus.cc:2592] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority W0918 20:39:41.444666 28359 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:35062: Illegal state: Replica 8654236fd4b14655929a78c02e6e8021 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.455744 28208 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:50378: Illegal state: Replica d32ba85bbbe247d7945b09267f723959 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.475488 28287 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:51164: Illegal state: Replica a68e2c30ad9f4733bbf8eb54594ce86c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.493911 28359 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:35062: Illegal state: Replica 8654236fd4b14655929a78c02e6e8021 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.509531 28208 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:50378: Illegal state: Replica d32ba85bbbe247d7945b09267f723959 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.511009 28480 log.cc:669] Time spent T 20884cc5471a4921b39674fa4462064c P 8654236fd4b14655929a78c02e6e8021: Append to log took a long time: real 0.086s user 0.000s sys 0.001s W0918 20:39:41.532533 28287 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:51164: Illegal state: Replica a68e2c30ad9f4733bbf8eb54594ce86c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.546754 28359 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:35062: Illegal state: Replica 8654236fd4b14655929a78c02e6e8021 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.562531 28208 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:50378: Illegal state: Replica d32ba85bbbe247d7945b09267f723959 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.590852 28287 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:51164: Illegal state: Replica a68e2c30ad9f4733bbf8eb54594ce86c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.609323 28359 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:35062: Illegal state: Replica 8654236fd4b14655929a78c02e6e8021 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.626978 28208 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:50378: Illegal state: Replica d32ba85bbbe247d7945b09267f723959 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.694824 28287 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:51164: Illegal state: Replica a68e2c30ad9f4733bbf8eb54594ce86c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.713826 28359 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:35062: Illegal state: Replica 8654236fd4b14655929a78c02e6e8021 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.734253 28208 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:50378: Illegal state: Replica d32ba85bbbe247d7945b09267f723959 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.762382 28287 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:51164: Illegal state: Replica a68e2c30ad9f4733bbf8eb54594ce86c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.785307 28359 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:35062: Illegal state: Replica 8654236fd4b14655929a78c02e6e8021 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.808748 28208 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:50378: Illegal state: Replica d32ba85bbbe247d7945b09267f723959 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.838270 28287 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:51164: Illegal state: Replica a68e2c30ad9f4733bbf8eb54594ce86c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.862134 28359 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:35062: Illegal state: Replica 8654236fd4b14655929a78c02e6e8021 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.888048 28208 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:50378: Illegal state: Replica d32ba85bbbe247d7945b09267f723959 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.922477 28287 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:51164: Illegal state: Replica a68e2c30ad9f4733bbf8eb54594ce86c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.951541 28359 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:35062: Illegal state: Replica 8654236fd4b14655929a78c02e6e8021 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:41.981446 28208 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:50378: Illegal state: Replica d32ba85bbbe247d7945b09267f723959 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:42.024684 28287 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:51164: Illegal state: Replica a68e2c30ad9f4733bbf8eb54594ce86c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:42.054996 28359 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:35062: Illegal state: Replica 8654236fd4b14655929a78c02e6e8021 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:42.088892 28208 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:50378: Illegal state: Replica d32ba85bbbe247d7945b09267f723959 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:42.131665 28287 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:51164: Illegal state: Replica a68e2c30ad9f4733bbf8eb54594ce86c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:42.167665 28359 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:35062: Illegal state: Replica 8654236fd4b14655929a78c02e6e8021 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:42.204646 28208 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:50378: Illegal state: Replica d32ba85bbbe247d7945b09267f723959 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:42.290238 28287 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:51164: Illegal state: Replica a68e2c30ad9f4733bbf8eb54594ce86c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:42.334018 28359 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:35062: Illegal state: Replica 8654236fd4b14655929a78c02e6e8021 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:42.372781 28208 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:50378: Illegal state: Replica d32ba85bbbe247d7945b09267f723959 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:42.414958 28287 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:51164: Illegal state: Replica a68e2c30ad9f4733bbf8eb54594ce86c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:42.453402 28359 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:35062: Illegal state: Replica 8654236fd4b14655929a78c02e6e8021 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:42.497109 28208 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:50378: Illegal state: Replica d32ba85bbbe247d7945b09267f723959 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:42.552961 28287 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:51164: Illegal state: Replica a68e2c30ad9f4733bbf8eb54594ce86c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:42.596128 28359 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:35062: Illegal state: Replica 8654236fd4b14655929a78c02e6e8021 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:42.640532 28208 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:50378: Illegal state: Replica d32ba85bbbe247d7945b09267f723959 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } W0918 20:39:42.694128 28287 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:51164: Illegal state: Replica a68e2c30ad9f4733bbf8eb54594ce86c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } I0918 20:39:42.701800 28490 raft_consensus.cc:472] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a68e2c30ad9f4733bbf8eb54594ce86c) I0918 20:39:42.702615 28490 raft_consensus.cc:494] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } I0918 20:39:42.711143 28490 leader_election.cc:296] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a68e2c30ad9f4733bbf8eb54594ce86c (127.18.222.2:46789), d32ba85bbbe247d7945b09267f723959 (127.18.222.1:38599) I0918 20:39:42.743139 28227 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0e7777829e0c46a3a5dc016b95c26a5b" candidate_uuid: "8654236fd4b14655929a78c02e6e8021" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d32ba85bbbe247d7945b09267f723959" is_pre_election: true I0918 20:39:42.744032 28227 raft_consensus.cc:2280] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8654236fd4b14655929a78c02e6e8021 in current term 2: Already voted for candidate d32ba85bbbe247d7945b09267f723959 in this term. W0918 20:39:42.743686 28359 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:35062: Illegal state: Replica 8654236fd4b14655929a78c02e6e8021 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } I0918 20:39:42.758141 28308 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0e7777829e0c46a3a5dc016b95c26a5b" candidate_uuid: "8654236fd4b14655929a78c02e6e8021" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" is_pre_election: true I0918 20:39:42.758726 28308 raft_consensus.cc:2350] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8654236fd4b14655929a78c02e6e8021 in term 2. I0918 20:39:42.760056 28340 leader_election.cc:310] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8654236fd4b14655929a78c02e6e8021, a68e2c30ad9f4733bbf8eb54594ce86c; no voters: d32ba85bbbe247d7945b09267f723959 I0918 20:39:42.761170 28490 raft_consensus.cc:2644] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0918 20:39:42.761560 28490 raft_consensus.cc:472] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a68e2c30ad9f4733bbf8eb54594ce86c) I0918 20:39:42.761937 28490 raft_consensus.cc:2890] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 1 FOLLOWER]: Advancing to term 2 I0918 20:39:42.769791 28490 raft_consensus.cc:494] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } I0918 20:39:42.771242 28490 leader_election.cc:296] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [CANDIDATE]: Term 2 election: Requested vote from peers a68e2c30ad9f4733bbf8eb54594ce86c (127.18.222.2:46789), d32ba85bbbe247d7945b09267f723959 (127.18.222.1:38599) I0918 20:39:42.772238 28227 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0e7777829e0c46a3a5dc016b95c26a5b" candidate_uuid: "8654236fd4b14655929a78c02e6e8021" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d32ba85bbbe247d7945b09267f723959" I0918 20:39:42.772655 28308 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0e7777829e0c46a3a5dc016b95c26a5b" candidate_uuid: "8654236fd4b14655929a78c02e6e8021" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" I0918 20:39:42.773245 28227 raft_consensus.cc:2280] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8654236fd4b14655929a78c02e6e8021 in current term 2: Already voted for candidate d32ba85bbbe247d7945b09267f723959 in this term. I0918 20:39:42.782611 28308 raft_consensus.cc:2350] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8654236fd4b14655929a78c02e6e8021 in term 2. I0918 20:39:42.784092 28340 leader_election.cc:310] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8654236fd4b14655929a78c02e6e8021, a68e2c30ad9f4733bbf8eb54594ce86c; no voters: d32ba85bbbe247d7945b09267f723959 I0918 20:39:42.786197 28490 raft_consensus.cc:2644] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 2 FOLLOWER]: Leader election won for term 2 I0918 20:39:42.788101 28490 raft_consensus.cc:667] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 2 LEADER]: Becoming Leader. State: Replica: 8654236fd4b14655929a78c02e6e8021, State: Running, Role: LEADER I0918 20:39:42.788767 28490 consensus_queue.cc:206] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } W0918 20:39:42.793663 28208 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:50378: Illegal state: Replica d32ba85bbbe247d7945b09267f723959 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } I0918 20:39:42.802597 28078 catalog_manager.cc:4131] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 reported cstate change: term changed from 1 to 2, leader changed from a68e2c30ad9f4733bbf8eb54594ce86c (127.18.222.2) to 8654236fd4b14655929a78c02e6e8021 (127.18.222.3). New cstate: current_term: 2 leader_uuid: "8654236fd4b14655929a78c02e6e8021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } health_report { overall_health: UNKNOWN } } } W0918 20:39:42.848579 28287 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:51164: Illegal state: Replica a68e2c30ad9f4733bbf8eb54594ce86c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } } ================== WARNING: ThreadSanitizer: data race (pid=19320) Write of size 8 at 0x7b0c0022e870 by thread T8 (mutexes: read M694252858918829320, write M607000): #0 operator delete(void*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 (ksck_remote-test+0x4db331) #1 __libcpp_deallocate /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/libcxx/include/new:236:3 (libc++.so.1+0xd7ab0) #2 deallocate /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/libcxx/include/memory:1796 (libc++.so.1+0xd7ab0) #3 deallocate /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/libcxx/include/memory:1555 (libc++.so.1+0xd7ab0) #4 std::__1::basic_string, std::__1::allocator >::~basic_string() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/libcxx/include/string:1941 (libc++.so.1+0xd7ab0) #5 google::protobuf::internal::ArenaStringPtr::DestroyNoArena(std::__1::basic_string, std::__1::allocator > const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/google/protobuf/arenastring.h:258:7 (libclient_proto.so+0x16e02) #6 kudu::consensus::RaftPeerPB::SharedDtor() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:1039:19 (libconsensus_metadata_proto.so+0x18a63) #7 kudu::consensus::RaftPeerPB::~RaftPeerPB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:1035:3 (libconsensus_metadata_proto.so+0x18a03) #8 kudu::consensus::RaftPeerPB::~RaftPeerPB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:1033:27 (libconsensus_metadata_proto.so+0x18b29) #9 google::protobuf::internal::GenericTypeHandler::Delete(kudu::consensus::RaftPeerPB*, google::protobuf::Arena*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/google/protobuf/repeated_field.h:615:7 (libconsensus.so+0xc0920) #10 void google::protobuf::internal::RepeatedPtrFieldBase::Destroy::TypeHandler>() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/google/protobuf/repeated_field.h:1429:7 (libconsensus.so+0xc08a9) #11 google::protobuf::RepeatedPtrField::~RepeatedPtrField() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/google/protobuf/repeated_field.h:1892:3 (libconsensus.so+0xbfaaa) #12 kudu::consensus::RaftConfigPB::~RaftConfigPB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:1699:1 (libconsensus_metadata_proto.so+0x19ea4) #13 kudu::consensus::RaftConfigPB::~RaftConfigPB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:1696:31 (libconsensus_metadata_proto.so+0x19ef9) #14 kudu::consensus::ConsensusStatePB::SharedDtor() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:2189:44 (libconsensus_metadata_proto.so+0x1afb0) #15 kudu::consensus::ConsensusStatePB::~ConsensusStatePB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:2184:3 (libconsensus_metadata_proto.so+0x1af23) #16 kudu::consensus::ConsensusStatePB::~ConsensusStatePB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:2182:39 (libconsensus_metadata_proto.so+0x1b019) #17 kudu::master::SysTabletsEntryPB::SharedDtor() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/master/master.pb.cc:3375:44 (libmaster_proto.so+0xdf114) #18 kudu::master::SysTabletsEntryPB::~SysTabletsEntryPB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/master/master.pb.cc:3366:3 (libmaster_proto.so+0xdf043) #19 kudu::master::PersistentTabletInfo::~PersistentTabletInfo() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.h:123:8 (libmaster.so+0x1610a9) #20 std::__1::default_delete::operator()(kudu::master::PersistentTabletInfo*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/memory:2285:5 (libmaster.so+0x135ce6) #21 std::__1::unique_ptr >::reset(kudu::master::PersistentTabletInfo*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/memory:2598 (libmaster.so+0x135ce6) #22 kudu::CowObject::CommitMutation() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/cow_object.h:93 (libmaster.so+0x135ce6) #23 kudu::CowGroupLock, std::__1::allocator >, kudu::master::PersistentTabletInfo>::Commit() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/cow_object.h:386:17 (libmaster.so+0x138081) #24 kudu::master::CatalogManager::ProcessTabletReport(kudu::master::TSDescriptor*, kudu::master::TabletReportPB const&, kudu::master::TabletReportUpdatesPB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:4277:16 (libmaster.so+0x126419) #25 kudu::master::MasterServiceImpl::TSHeartbeat(kudu::master::TSHeartbeatRequestPB const*, kudu::master::TSHeartbeatResponsePB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/master_service.cc:259:44 (libmaster.so+0x19e650) #26 kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_1::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/master/master.service.cc:151:13 (libmaster_proto.so+0x165854) #27 _ZNSt3__18__invokeIRZN4kudu6master15MasterServiceIfC1ERK13scoped_refptrINS1_12MetricEntityEERKS4_INS1_3rpc13ResultTrackerEEE3$_1JPKN6google8protobuf7MessageEPSI_PNS9_10RpcContextEEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSO_DpOSP_ /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/type_traits:4482:1 (libmaster_proto.so+0x1657e1) #28 void std::__1::__invoke_void_return_wrapper::__call const&, scoped_refptr const&)::$_1&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*>(kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_1&, google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/__functional_base:349 (libmaster_proto.so+0x1657e1) #29 std::__1::__function::__func const&, scoped_refptr const&)::$_1, std::__1::allocator const&, scoped_refptr const&)::$_1>, void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1562:12 (libmaster_proto.so+0x1656f4) #30 std::__1::function::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1916:12 (libkrpc.so+0xf2f29) #31 kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_if.cc:139:3 (libkrpc.so+0xf29e5) #32 kudu::rpc::ServicePool::RunThread() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_pool.cc:225:15 (libkrpc.so+0xf3dcc) #33 boost::_mfi::mf0::operator()(kudu::rpc::ServicePool*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xf5af9) #34 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xf5a4a) #35 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xf59d3) #36 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xf57c9) #37 boost::function0::operator()() const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xba0b1) #38 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.cc:657:3 (libkudu_util.so+0x1ee174) Previous read of size 8 at 0x7b0c0022e870 by thread T433 (mutexes: read M694252858918829320): #0 strings::memeq(void const*, void const*, unsigned long) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/gutil/strings/fastmem.h:46:36 (libitest_util.so+0x5eb6a) #1 operator==(StringPiece, StringPiece) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/gutil/strings/stringpiece.h:314:7 (libitest_util.so+0x5eaa1) #2 std::__1::equal_to::operator()(StringPiece const&, StringPiece const&) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:670:21 (libmaster.so+0x159867) #3 google::dense_hashtable, StringPiece, GoodFastHash, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SelectKey, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SetKey, std::__1::equal_to, google::libc_allocator_with_realloc > >::KeyInfo::equals(StringPiece const&, StringPiece const&) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/common/include/sparsehash/internal/densehashtable.h:1287 (libmaster.so+0x159867) #4 google::dense_hashtable, StringPiece, GoodFastHash, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SelectKey, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SetKey, std::__1::equal_to, google::libc_allocator_with_realloc > >::equals(StringPiece const&, StringPiece const&) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/common/include/sparsehash/internal/densehashtable.h:1299:21 (libmaster.so+0x1596dd) #5 google::dense_hashtable, StringPiece, GoodFastHash, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SelectKey, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SetKey, std::__1::equal_to, google::libc_allocator_with_realloc > >::find_position(StringPiece const&) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/common/include/sparsehash/internal/densehashtable.h:879:18 (libmaster.so+0x171b9b) #6 google::dense_hashtable, StringPiece, GoodFastHash, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SelectKey, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SetKey, std::__1::equal_to, google::libc_allocator_with_realloc > >::find(StringPiece const&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/common/include/sparsehash/internal/densehashtable.h:892:43 (libmaster.so+0x171974) #7 google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::find(StringPiece const&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/common/include/sparsehash/dense_hash_map:249:51 (libmaster.so+0x171809) #8 std::__1::pair, std::__1::equal_to, google::libc_allocator_with_realloc > >::mapped_type* const, bool> ComputeIfAbsentReturnAbsense, std::__1::equal_to, google::libc_allocator_with_realloc > >, kudu::master::CatalogManager::BuildLocationsForTablet(scoped_refptr const&, kudu::master::ReplicaTypeFilter, kudu::master::TabletLocationsPB*, kudu::master::CatalogManager::TSInfosDict*)::$_17>(google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >*, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::key_type const&, kudu::master::CatalogManager::BuildLocationsForTablet(scoped_refptr const&, kudu::master::ReplicaTypeFilter, kudu::master::TabletLocationsPB*, kudu::master::CatalogManager::TSInfosDict*)::$_17) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/gutil/map-util.h:883:53 (libmaster.so+0x13048a) #9 google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::mapped_type* const ComputeIfAbsent, std::__1::equal_to, google::libc_allocator_with_realloc > >, kudu::master::CatalogManager::BuildLocationsForTablet(scoped_refptr const&, kudu::master::ReplicaTypeFilter, kudu::master::TabletLocationsPB*, kudu::master::CatalogManager::TSInfosDict*)::$_17>(google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >*, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::key_type const&, kudu::master::CatalogManager::BuildLocationsForTablet(scoped_refptr const&, kudu::master::ReplicaTypeFilter, kudu::master::TabletLocationsPB*, kudu::master::CatalogManager::TSInfosDict*)::$_17) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/gutil/map-util.h:906:10 (libmaster.so+0x12a4c0) #10 kudu::master::CatalogManager::BuildLocationsForTablet(scoped_refptr const&, kudu::master::ReplicaTypeFilter, kudu::master::TabletLocationsPB*, kudu::master::CatalogManager::TSInfosDict*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:4846:18 (libmaster.so+0x12a075) #11 kudu::master::CatalogManager::GetTableLocations(kudu::master::GetTableLocationsRequestPB const*, kudu::master::GetTableLocationsResponsePB*, boost::optional, std::__1::allocator > const&>) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:5028:16 (libmaster.so+0x12b888) #12 kudu::master::MasterServiceImpl::GetTableLocations(kudu::master::GetTableLocationsRequestPB const*, kudu::master::GetTableLocationsResponsePB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/master_service.cc:441:42 (libmaster.so+0x19fcf5) #13 kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_21::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/master/master.service.cc:351:13 (libmaster_proto.so+0x169f97) #14 _ZNSt3__18__invokeIRZN4kudu6master15MasterServiceIfC1ERK13scoped_refptrINS1_12MetricEntityEERKS4_INS1_3rpc13ResultTrackerEEE4$_21JPKN6google8protobuf7MessageEPSI_PNS9_10RpcContextEEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSO_DpOSP_ /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/type_traits:4482:1 (libmaster_proto.so+0x169f21) #15 void std::__1::__invoke_void_return_wrapper::__call const&, scoped_refptr const&)::$_21&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*>(kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_21&, google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/__functional_base:349 (libmaster_proto.so+0x169f21) #16 std::__1::__function::__func const&, scoped_refptr const&)::$_21, std::__1::allocator const&, scoped_refptr const&)::$_21>, void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1562:12 (libmaster_proto.so+0x169e34) #17 std::__1::function::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1916:12 (libkrpc.so+0xf2f29) #18 kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_if.cc:139:3 (libkrpc.so+0xf29e5) #19 kudu::rpc::ServicePool::RunThread() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_pool.cc:225:15 (libkrpc.so+0xf3dcc) #20 boost::_mfi::mf0::operator()(kudu::rpc::ServicePool*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xf5af9) #21 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xf5a4a) #22 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xf59d3) #23 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xf57c9) #24 boost::function0::operator()() const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xba0b1) #25 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.cc:657:3 (libkudu_util.so+0x1ee174) Mutex M694252858918829320 is already destroyed. Mutex M607000 (0x7b6800108028) created at: #0 pthread_mutex_init /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1211 (ksck_remote-test+0x45dcb5) #1 kudu::Mutex::Mutex() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/mutex.cc:65:8 (libkudu_util.so+0x1b5f6f) #2 kudu::RWCLock::RWCLock() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/rwc_lock.cc:30:10 (libkudu_util.so+0x1dd16d) #3 kudu::CowObject::CowObject() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/cow_object.h:40:3 (libmaster.so+0x14139a) #4 kudu::master::TabletInfo::TabletInfo(scoped_refptr, std::__1::basic_string, std::__1::allocator >) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:5302:13 (libmaster.so+0x12d995) #5 kudu::master::CatalogManager::CreateTabletInfo(scoped_refptr const&, kudu::PartitionPB const&, boost::optional, std::__1::allocator > > const&) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:1822:40 (libmaster.so+0x111ae4) #6 kudu::master::CatalogManager::CreateTable(kudu::master::CreateTableRequestPB const*, kudu::master::CreateTableResponsePB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:1676:26 (libmaster.so+0x10f90b) #7 kudu::master::MasterServiceImpl::CreateTable(kudu::master::CreateTableRequestPB const*, kudu::master::CreateTableResponsePB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/master_service.cc:341:42 (libmaster.so+0x19f1ef) #8 kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_7::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/master/master.service.cc:211:13 (libmaster_proto.so+0x166db4) #9 _ZNSt3__18__invokeIRZN4kudu6master15MasterServiceIfC1ERK13scoped_refptrINS1_12MetricEntityEERKS4_INS1_3rpc13ResultTrackerEEE3$_7JPKN6google8protobuf7MessageEPSI_PNS9_10RpcContextEEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSO_DpOSP_ /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/type_traits:4482:1 (libmaster_proto.so+0x166d41) #10 void std::__1::__invoke_void_return_wrapper::__call const&, scoped_refptr const&)::$_7&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*>(kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_7&, google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/__functional_base:349 (libmaster_proto.so+0x166d41) #11 std::__1::__function::__func const&, scoped_refptr const&)::$_7, std::__1::allocator const&, scoped_refptr const&)::$_7>, void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1562:12 (libmaster_proto.so+0x166c54) #12 std::__1::function::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1916:12 (libkrpc.so+0xf2f29) #13 kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_if.cc:139:3 (libkrpc.so+0xf29e5) #14 kudu::rpc::ServicePool::RunThread() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_pool.cc:225:15 (libkrpc.so+0xf3dcc) #15 boost::_mfi::mf0::operator()(kudu::rpc::ServicePool*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xf5af9) #16 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xf5a4a) #17 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xf59d3) #18 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xf57c9) #19 boost::function0::operator()() const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xba0b1) #20 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.cc:657:3 (libkudu_util.so+0x1ee174) Thread T8 'rpc worker-2807' (tid=28078, running) created by main thread at: #0 pthread_create /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (ksck_remote-test+0x44b806) #1 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.cc:601:15 (libkudu_util.so+0x1ed95b) #2 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::rpc::ServicePool::* const&)(), kudu::rpc::ServicePool* const&, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.h:164:12 (libkrpc.so+0xf4ee5) #3 kudu::rpc::ServicePool::Init(int) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_pool.cc:92:5 (libkrpc.so+0xf39d6) #4 kudu::RpcServer::RegisterService(gscoped_ptr >) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/server/rpc_server.cc:157:3 (libserver_process.so+0x83529) #5 kudu::server::ServerBase::RegisterService(gscoped_ptr >) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/server/server_base.cc:638:23 (libserver_process.so+0x8a0ff) #6 kudu::master::Master::StartAsync() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/master.cc:186:3 (libmaster.so+0x18830f) #7 kudu::master::MiniMaster::Start() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/mini_master.cc:94:3 (libmaster.so+0x1a7e64) #8 kudu::cluster::InternalMiniCluster::StartMasters() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/mini-cluster/internal_mini_cluster.cc:160:5 (libmini_cluster.so+0x42bf3) #9 kudu::cluster::InternalMiniCluster::Start() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/mini-cluster/internal_mini_cluster.cc:99:3 (libmini_cluster.so+0x426cd) #10 kudu::tools::RemoteKsckTest::SetUp() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/tools/ksck_remote-test.cc:123:5 (ksck_remote-test+0x4ef737) #11 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef) #12 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef) #13 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2470:3 (libgmock.so+0x343c1) #14 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c) #15 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226) #16 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa) #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f) #18 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f) #19 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2) #20 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x351b) #21 main /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2cc6) Thread T433 'rpc worker-2808' (tid=28080, running) created by main thread at: #0 pthread_create /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (ksck_remote-test+0x44b806) #1 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.cc:601:15 (libkudu_util.so+0x1ed95b) #2 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::rpc::ServicePool::* const&)(), kudu::rpc::ServicePool* const&, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.h:164:12 (libkrpc.so+0xf4ee5) #3 kudu::rpc::ServicePool::Init(int) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_pool.cc:92:5 (libkrpc.so+0xf39d6) #4 kudu::RpcServer::RegisterService(gscoped_ptr >) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/server/rpc_server.cc:157:3 (libserver_process.so+0x83529) #5 kudu::server::ServerBase::RegisterService(gscoped_ptr >) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/server/server_base.cc:638:23 (libserver_process.so+0x8a0ff) #6 kudu::master::Master::StartAsync() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/master.cc:186:3 (libmaster.so+0x18830f) #7 kudu::master::MiniMaster::Start() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/mini_master.cc:94:3 (libmaster.so+0x1a7e64) #8 kudu::cluster::InternalMiniCluster::StartMasters() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/mini-cluster/internal_mini_cluster.cc:160:5 (libmini_cluster.so+0x42bf3) #9 kudu::cluster::InternalMiniCluster::Start() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/mini-cluster/internal_mini_cluster.cc:99:3 (libmini_cluster.so+0x426cd) #10 kudu::tools::RemoteKsckTest::SetUp() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/tools/ksck_remote-test.cc:123:5 (ksck_remote-test+0x4ef737) #11 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef) #12 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef) #13 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2470:3 (libgmock.so+0x343c1) #14 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c) #15 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226) #16 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa) #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f) #18 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f) #19 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2) #20 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x351b) #21 main /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2cc6) SUMMARY: ThreadSanitizer: data race /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 in operator delete(void*) ================== ================== WARNING: ThreadSanitizer: data race (pid=19320) Write of size 8 at 0x7b0c0022e878 by thread T8 (mutexes: read M694252858918829320, write M607000): #0 operator delete(void*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 (ksck_remote-test+0x4db331) #1 __libcpp_deallocate /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/libcxx/include/new:236:3 (libc++.so.1+0xd7ab0) #2 deallocate /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/libcxx/include/memory:1796 (libc++.so.1+0xd7ab0) #3 deallocate /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/libcxx/include/memory:1555 (libc++.so.1+0xd7ab0) #4 std::__1::basic_string, std::__1::allocator >::~basic_string() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/libcxx/include/string:1941 (libc++.so.1+0xd7ab0) #5 google::protobuf::internal::ArenaStringPtr::DestroyNoArena(std::__1::basic_string, std::__1::allocator > const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/google/protobuf/arenastring.h:258:7 (libclient_proto.so+0x16e02) #6 kudu::consensus::RaftPeerPB::SharedDtor() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:1039:19 (libconsensus_metadata_proto.so+0x18a63) #7 kudu::consensus::RaftPeerPB::~RaftPeerPB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:1035:3 (libconsensus_metadata_proto.so+0x18a03) #8 kudu::consensus::RaftPeerPB::~RaftPeerPB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:1033:27 (libconsensus_metadata_proto.so+0x18b29) #9 google::protobuf::internal::GenericTypeHandler::Delete(kudu::consensus::RaftPeerPB*, google::protobuf::Arena*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/google/protobuf/repeated_field.h:615:7 (libconsensus.so+0xc0920) #10 void google::protobuf::internal::RepeatedPtrFieldBase::Destroy::TypeHandler>() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/google/protobuf/repeated_field.h:1429:7 (libconsensus.so+0xc08a9) #11 google::protobuf::RepeatedPtrField::~RepeatedPtrField() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/google/protobuf/repeated_field.h:1892:3 (libconsensus.so+0xbfaaa) #12 kudu::consensus::RaftConfigPB::~RaftConfigPB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:1699:1 (libconsensus_metadata_proto.so+0x19ea4) #13 kudu::consensus::RaftConfigPB::~RaftConfigPB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:1696:31 (libconsensus_metadata_proto.so+0x19ef9) #14 kudu::consensus::ConsensusStatePB::SharedDtor() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:2189:44 (libconsensus_metadata_proto.so+0x1afb0) #15 kudu::consensus::ConsensusStatePB::~ConsensusStatePB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:2184:3 (libconsensus_metadata_proto.so+0x1af23) #16 kudu::consensus::ConsensusStatePB::~ConsensusStatePB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:2182:39 (libconsensus_metadata_proto.so+0x1b019) #17 kudu::master::SysTabletsEntryPB::SharedDtor() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/master/master.pb.cc:3375:44 (libmaster_proto.so+0xdf114) #18 kudu::master::SysTabletsEntryPB::~SysTabletsEntryPB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/master/master.pb.cc:3366:3 (libmaster_proto.so+0xdf043) #19 kudu::master::PersistentTabletInfo::~PersistentTabletInfo() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.h:123:8 (libmaster.so+0x1610a9) #20 std::__1::default_delete::operator()(kudu::master::PersistentTabletInfo*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/memory:2285:5 (libmaster.so+0x135ce6) #21 std::__1::unique_ptr >::reset(kudu::master::PersistentTabletInfo*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/memory:2598 (libmaster.so+0x135ce6) #22 kudu::CowObject::CommitMutation() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/cow_object.h:93 (libmaster.so+0x135ce6) #23 kudu::CowGroupLock, std::__1::allocator >, kudu::master::PersistentTabletInfo>::Commit() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/cow_object.h:386:17 (libmaster.so+0x138081) #24 kudu::master::CatalogManager::ProcessTabletReport(kudu::master::TSDescriptor*, kudu::master::TabletReportPB const&, kudu::master::TabletReportUpdatesPB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:4277:16 (libmaster.so+0x126419) #25 kudu::master::MasterServiceImpl::TSHeartbeat(kudu::master::TSHeartbeatRequestPB const*, kudu::master::TSHeartbeatResponsePB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/master_service.cc:259:44 (libmaster.so+0x19e650) #26 kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_1::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/master/master.service.cc:151:13 (libmaster_proto.so+0x165854) #27 _ZNSt3__18__invokeIRZN4kudu6master15MasterServiceIfC1ERK13scoped_refptrINS1_12MetricEntityEERKS4_INS1_3rpc13ResultTrackerEEE3$_1JPKN6google8protobuf7MessageEPSI_PNS9_10RpcContextEEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSO_DpOSP_ /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/type_traits:4482:1 (libmaster_proto.so+0x1657e1) #28 void std::__1::__invoke_void_return_wrapper::__call const&, scoped_refptr const&)::$_1&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*>(kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_1&, google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/__functional_base:349 (libmaster_proto.so+0x1657e1) #29 std::__1::__function::__func const&, scoped_refptr const&)::$_1, std::__1::allocator const&, scoped_refptr const&)::$_1>, void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1562:12 (libmaster_proto.so+0x1656f4) #30 std::__1::function::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1916:12 (libkrpc.so+0xf2f29) #31 kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_if.cc:139:3 (libkrpc.so+0xf29e5) #32 kudu::rpc::ServicePool::RunThread() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_pool.cc:225:15 (libkrpc.so+0xf3dcc) #33 boost::_mfi::mf0::operator()(kudu::rpc::ServicePool*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xf5af9) #34 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xf5a4a) #35 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xf59d3) #36 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xf57c9) #37 boost::function0::operator()() const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xba0b1) #38 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.cc:657:3 (libkudu_util.so+0x1ee174) Previous read of size 8 at 0x7b0c0022e878 by thread T433 (mutexes: read M694252858918829320): #0 strings::memeq(void const*, void const*, unsigned long) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/gutil/strings/fastmem.h:61:38 (libitest_util.so+0x5ec01) #1 operator==(StringPiece, StringPiece) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/gutil/strings/stringpiece.h:314:7 (libitest_util.so+0x5eaa1) #2 std::__1::equal_to::operator()(StringPiece const&, StringPiece const&) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:670:21 (libmaster.so+0x159867) #3 google::dense_hashtable, StringPiece, GoodFastHash, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SelectKey, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SetKey, std::__1::equal_to, google::libc_allocator_with_realloc > >::KeyInfo::equals(StringPiece const&, StringPiece const&) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/common/include/sparsehash/internal/densehashtable.h:1287 (libmaster.so+0x159867) #4 google::dense_hashtable, StringPiece, GoodFastHash, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SelectKey, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SetKey, std::__1::equal_to, google::libc_allocator_with_realloc > >::equals(StringPiece const&, StringPiece const&) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/common/include/sparsehash/internal/densehashtable.h:1299:21 (libmaster.so+0x1596dd) #5 google::dense_hashtable, StringPiece, GoodFastHash, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SelectKey, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SetKey, std::__1::equal_to, google::libc_allocator_with_realloc > >::find_position(StringPiece const&) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/common/include/sparsehash/internal/densehashtable.h:879:18 (libmaster.so+0x171b9b) #6 google::dense_hashtable, StringPiece, GoodFastHash, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SelectKey, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SetKey, std::__1::equal_to, google::libc_allocator_with_realloc > >::find(StringPiece const&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/common/include/sparsehash/internal/densehashtable.h:892:43 (libmaster.so+0x171974) #7 google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::find(StringPiece const&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/common/include/sparsehash/dense_hash_map:249:51 (libmaster.so+0x171809) #8 std::__1::pair, std::__1::equal_to, google::libc_allocator_with_realloc > >::mapped_type* const, bool> ComputeIfAbsentReturnAbsense, std::__1::equal_to, google::libc_allocator_with_realloc > >, kudu::master::CatalogManager::BuildLocationsForTablet(scoped_refptr const&, kudu::master::ReplicaTypeFilter, kudu::master::TabletLocationsPB*, kudu::master::CatalogManager::TSInfosDict*)::$_17>(google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >*, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::key_type const&, kudu::master::CatalogManager::BuildLocationsForTablet(scoped_refptr const&, kudu::master::ReplicaTypeFilter, kudu::master::TabletLocationsPB*, kudu::master::CatalogManager::TSInfosDict*)::$_17) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/gutil/map-util.h:883:53 (libmaster.so+0x13048a) #9 google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::mapped_type* const ComputeIfAbsent, std::__1::equal_to, google::libc_allocator_with_realloc > >, kudu::master::CatalogManager::BuildLocationsForTablet(scoped_refptr const&, kudu::master::ReplicaTypeFilter, kudu::master::TabletLocationsPB*, kudu::master::CatalogManager::TSInfosDict*)::$_17>(google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >*, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::key_type const&, kudu::master::CatalogManager::BuildLocationsForTablet(scoped_refptr const&, kudu::master::ReplicaTypeFilter, kudu::master::TabletLocationsPB*, kudu::master::CatalogManager::TSInfosDict*)::$_17) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/gutil/map-util.h:906:10 (libmaster.so+0x12a4c0) #10 kudu::master::CatalogManager::BuildLocationsForTablet(scoped_refptr const&, kudu::master::ReplicaTypeFilter, kudu::master::TabletLocationsPB*, kudu::master::CatalogManager::TSInfosDict*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:4846:18 (libmaster.so+0x12a075) #11 kudu::master::CatalogManager::GetTableLocations(kudu::master::GetTableLocationsRequestPB const*, kudu::master::GetTableLocationsResponsePB*, boost::optional, std::__1::allocator > const&>) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:5028:16 (libmaster.so+0x12b888) #12 kudu::master::MasterServiceImpl::GetTableLocations(kudu::master::GetTableLocationsRequestPB const*, kudu::master::GetTableLocationsResponsePB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/master_service.cc:441:42 (libmaster.so+0x19fcf5) #13 kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_21::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/master/master.service.cc:351:13 (libmaster_proto.so+0x169f97) #14 _ZNSt3__18__invokeIRZN4kudu6master15MasterServiceIfC1ERK13scoped_refptrINS1_12MetricEntityEERKS4_INS1_3rpc13ResultTrackerEEE4$_21JPKN6google8protobuf7MessageEPSI_PNS9_10RpcContextEEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSO_DpOSP_ /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/type_traits:4482:1 (libmaster_proto.so+0x169f21) #15 void std::__1::__invoke_void_return_wrapper::__call const&, scoped_refptr const&)::$_21&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*>(kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_21&, google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/__functional_base:349 (libmaster_proto.so+0x169f21) #16 std::__1::__function::__func const&, scoped_refptr const&)::$_21, std::__1::allocator const&, scoped_refptr const&)::$_21>, void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1562:12 (libmaster_proto.so+0x169e34) #17 std::__1::function::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1916:12 (libkrpc.so+0xf2f29) #18 kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_if.cc:139:3 (libkrpc.so+0xf29e5) #19 kudu::rpc::ServicePool::RunThread() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_pool.cc:225:15 (libkrpc.so+0xf3dcc) #20 boost::_mfi::mf0::operator()(kudu::rpc::ServicePool*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xf5af9) #21 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xf5a4a) #22 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xf59d3) #23 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xf57c9) #24 boost::function0::operator()() const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xba0b1) #25 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.cc:657:3 (libkudu_util.so+0x1ee174) Mutex M694252858918829320 is already destroyed. Mutex M607000 (0x7b6800108028) created at: #0 pthread_mutex_init /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1211 (ksck_remote-test+0x45dcb5) #1 kudu::Mutex::Mutex() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/mutex.cc:65:8 (libkudu_util.so+0x1b5f6f) #2 kudu::RWCLock::RWCLock() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/rwc_lock.cc:30:10 (libkudu_util.so+0x1dd16d) #3 kudu::CowObject::CowObject() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/cow_object.h:40:3 (libmaster.so+0x14139a) #4 kudu::master::TabletInfo::TabletInfo(scoped_refptr, std::__1::basic_string, std::__1::allocator >) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:5302:13 (libmaster.so+0x12d995) #5 kudu::master::CatalogManager::CreateTabletInfo(scoped_refptr const&, kudu::PartitionPB const&, boost::optional, std::__1::allocator > > const&) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:1822:40 (libmaster.so+0x111ae4) #6 kudu::master::CatalogManager::CreateTable(kudu::master::CreateTableRequestPB const*, kudu::master::CreateTableResponsePB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:1676:26 (libmaster.so+0x10f90b) #7 kudu::master::MasterServiceImpl::CreateTable(kudu::master::CreateTableRequestPB const*, kudu::master::CreateTableResponsePB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/master_service.cc:341:42 (libmaster.so+0x19f1ef) #8 kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_7::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/master/master.service.cc:211:13 (libmaster_proto.so+0x166db4) #9 _ZNSt3__18__invokeIRZN4kudu6master15MasterServiceIfC1ERK13scoped_refptrINS1_12MetricEntityEERKS4_INS1_3rpc13ResultTrackerEEE3$_7JPKN6google8protobuf7MessageEPSI_PNS9_10RpcContextEEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSO_DpOSP_ /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/type_traits:4482:1 (libmaster_proto.so+0x166d41) #10 void std::__1::__invoke_void_return_wrapper::__call const&, scoped_refptr const&)::$_7&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*>(kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_7&, google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/__functional_base:349 (libmaster_proto.so+0x166d41) #11 std::__1::__function::__func const&, scoped_refptr const&)::$_7, std::__1::allocator const&, scoped_refptr const&)::$_7>, void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1562:12 (libmaster_proto.so+0x166c54) #12 std::__1::function::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1916:12 (libkrpc.so+0xf2f29) #13 kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_if.cc:139:3 (libkrpc.so+0xf29e5) #14 kudu::rpc::ServicePool::RunThread() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_pool.cc:225:15 (libkrpc.so+0xf3dcc) #15 boost::_mfi::mf0::operator()(kudu::rpc::ServicePool*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xf5af9) #16 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xf5a4a) #17 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xf59d3) #18 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xf57c9) #19 boost::function0::operator()() const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xba0b1) #20 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.cc:657:3 (libkudu_util.so+0x1ee174) Thread T8 'rpc worker-2807' (tid=28078, running) created by main thread at: #0 pthread_create /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (ksck_remote-test+0x44b806) #1 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.cc:601:15 (libkudu_util.so+0x1ed95b) #2 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::rpc::ServicePool::* const&)(), kudu::rpc::ServicePool* const&, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.h:164:12 (libkrpc.so+0xf4ee5) #3 kudu::rpc::ServicePool::Init(int) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_pool.cc:92:5 (libkrpc.so+0xf39d6) #4 kudu::RpcServer::RegisterService(gscoped_ptr >) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/server/rpc_server.cc:157:3 (libserver_process.so+0x83529) #5 kudu::server::ServerBase::RegisterService(gscoped_ptr >) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/server/server_base.cc:638:23 (libserver_process.so+0x8a0ff) #6 kudu::master::Master::StartAsync() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/master.cc:186:3 (libmaster.so+0x18830f) #7 kudu::master::MiniMaster::Start() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/mini_master.cc:94:3 (libmaster.so+0x1a7e64) #8 kudu::cluster::InternalMiniCluster::StartMasters() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/mini-cluster/internal_mini_cluster.cc:160:5 (libmini_cluster.so+0x42bf3) #9 kudu::cluster::InternalMiniCluster::Start() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/mini-cluster/internal_mini_cluster.cc:99:3 (libmini_cluster.so+0x426cd) #10 kudu::tools::RemoteKsckTest::SetUp() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/tools/ksck_remote-test.cc:123:5 (ksck_remote-test+0x4ef737) #11 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef) #12 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef) #13 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2470:3 (libgmock.so+0x343c1) #14 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c) #15 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226) #16 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa) #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f) #18 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f) #19 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2) #20 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x351b) #21 main /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2cc6) Thread T433 'rpc worker-2808' (tid=28080, running) created by main thread at: #0 pthread_create /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (ksck_remote-test+0x44b806) #1 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.cc:601:15 (libkudu_util.so+0x1ed95b) #2 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::rpc::ServicePool::* const&)(), kudu::rpc::ServicePool* const&, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.h:164:12 (libkrpc.so+0xf4ee5) #3 kudu::rpc::ServicePool::Init(int) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_pool.cc:92:5 (libkrpc.so+0xf39d6) #4 kudu::RpcServer::RegisterService(gscoped_ptr >) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/server/rpc_server.cc:157:3 (libserver_process.so+0x83529) #5 kudu::server::ServerBase::RegisterService(gscoped_ptr >) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/server/server_base.cc:638:23 (libserver_process.so+0x8a0ff) #6 kudu::master::Master::StartAsync() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/master.cc:186:3 (libmaster.so+0x18830f) #7 kudu::master::MiniMaster::Start() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/mini_master.cc:94:3 (libmaster.so+0x1a7e64) #8 kudu::cluster::InternalMiniCluster::StartMasters() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/mini-cluster/internal_mini_cluster.cc:160:5 (libmini_cluster.so+0x42bf3) #9 kudu::cluster::InternalMiniCluster::Start() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/mini-cluster/internal_mini_cluster.cc:99:3 (libmini_cluster.so+0x426cd) #10 kudu::tools::RemoteKsckTest::SetUp() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/tools/ksck_remote-test.cc:123:5 (ksck_remote-test+0x4ef737) #11 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef) #12 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef) #13 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2470:3 (libgmock.so+0x343c1) #14 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c) #15 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226) #16 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa) #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f) #18 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f) #19 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2) #20 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x351b) #21 main /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2cc6) SUMMARY: ThreadSanitizer: data race /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 in operator delete(void*) ================== I0918 20:39:43.055128 28479 raft_consensus.cc:472] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:43.055686 28479 raft_consensus.cc:494] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } I0918 20:39:43.057757 28479 leader_election.cc:296] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a68e2c30ad9f4733bbf8eb54594ce86c (127.18.222.2:46789), 8654236fd4b14655929a78c02e6e8021 (127.18.222.3:41335) I0918 20:39:43.059680 28308 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0e7777829e0c46a3a5dc016b95c26a5b" candidate_uuid: "d32ba85bbbe247d7945b09267f723959" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" is_pre_election: true I0918 20:39:43.060533 28308 raft_consensus.cc:2350] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d32ba85bbbe247d7945b09267f723959 in term 2. I0918 20:39:43.059479 28377 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0e7777829e0c46a3a5dc016b95c26a5b" candidate_uuid: "d32ba85bbbe247d7945b09267f723959" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8654236fd4b14655929a78c02e6e8021" is_pre_election: true I0918 20:39:43.062453 28377 raft_consensus.cc:2311] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate d32ba85bbbe247d7945b09267f723959 for term 3 because replica is either leader or believes a valid leader to be alive. I0918 20:39:43.063650 28186 leader_election.cc:310] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a68e2c30ad9f4733bbf8eb54594ce86c, d32ba85bbbe247d7945b09267f723959; no voters: I0918 20:39:43.065037 28479 raft_consensus.cc:2644] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0918 20:39:43.065461 28479 raft_consensus.cc:472] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:43.065747 28479 raft_consensus.cc:2890] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 2 FOLLOWER]: Advancing to term 3 ================== WARNING: ThreadSanitizer: data race (pid=19320) Write of size 8 at 0x7b0c0022e880 by thread T8 (mutexes: read M694252858918829320, write M607000): I0918 20:39:43.071082 28479 raft_consensus.cc:494] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } I0918 20:39:43.072844 28479 leader_election.cc:296] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [CANDIDATE]: Term 3 election: Requested vote from peers a68e2c30ad9f4733bbf8eb54594ce86c (127.18.222.2:46789), 8654236fd4b14655929a78c02e6e8021 (127.18.222.3:41335) #0 operator delete(void*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 (ksck_remote-test+0x4db331) #1 __libcpp_deallocate /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/libcxx/include/new:236:3 (libc++.so.1+0xd7ab0) #2 deallocate /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/libcxx/include/memory:1796 (libc++.so.1+0xd7ab0) #3 deallocate /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/libcxx/include/memory:1555 (libc++.so.1+0xd7ab0) I0918 20:39:43.074049 28377 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0e7777829e0c46a3a5dc016b95c26a5b" candidate_uuid: "d32ba85bbbe247d7945b09267f723959" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8654236fd4b14655929a78c02e6e8021" I0918 20:39:43.074934 28377 raft_consensus.cc:2311] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 2 LEADER]: Leader election vote request: Denying vote to candidate d32ba85bbbe247d7945b09267f723959 for term 3 because replica is either leader or believes a valid leader to be alive. I0918 20:39:43.074532 28308 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0e7777829e0c46a3a5dc016b95c26a5b" candidate_uuid: "d32ba85bbbe247d7945b09267f723959" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" I0918 20:39:43.075963 28308 raft_consensus.cc:2890] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 2 FOLLOWER]: Advancing to term 3 #4 std::__1::basic_string, std::__1::allocator >::~basic_string() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/libcxx/include/string:1941 (libc++.so.1+0xd7ab0) #5 google::protobuf::internal::ArenaStringPtr::DestroyNoArena(std::__1::basic_string, std::__1::allocator > const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/google/protobuf/arenastring.h:258:7 (libclient_proto.so+0x16e02) #6 kudu::consensus::RaftPeerPB::SharedDtor() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:1039:19 (libconsensus_metadata_proto.so+0x18a63) #7 kudu::consensus::RaftPeerPB::~RaftPeerPB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:1035:3 (libconsensus_metadata_proto.so+0x18a03) #8 kudu::consensus::RaftPeerPB::~RaftPeerPB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:1033:27 (libconsensus_metadata_proto.so+0x18b29) #9 google::protobuf::internal::GenericTypeHandler::Delete(kudu::consensus::RaftPeerPB*, google::protobuf::Arena*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/google/protobuf/repeated_field.h:615:7 (libconsensus.so+0xc0920) #10 void google::protobuf::internal::RepeatedPtrFieldBase::Destroy::TypeHandler>() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/google/protobuf/repeated_field.h:1429:7 (libconsensus.so+0xc08a9) #11 google::protobuf::RepeatedPtrField::~RepeatedPtrField() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/google/protobuf/repeated_field.h:1892:3 (libconsensus.so+0xbfaaa) #12 kudu::consensus::RaftConfigPB::~RaftConfigPB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:1699:1 (libconsensus_metadata_proto.so+0x19ea4) #13 kudu::consensus::RaftConfigPB::~RaftConfigPB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:1696:31 (libconsensus_metadata_proto.so+0x19ef9) #14 kudu::consensus::ConsensusStatePB::SharedDtor() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:2189:44 (libconsensus_metadata_proto.so+0x1afb0) #15 kudu::consensus::ConsensusStatePB::~ConsensusStatePB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:2184:3 (libconsensus_metadata_proto.so+0x1af23) #16 kudu::consensus::ConsensusStatePB::~ConsensusStatePB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:2182:39 (libconsensus_metadata_proto.so+0x1b019) #17 kudu::master::SysTabletsEntryPB::SharedDtor() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/master/master.pb.cc:3375:44 (libmaster_proto.so+0xdf114) #18 kudu::master::SysTabletsEntryPB::~SysTabletsEntryPB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/master/master.pb.cc:3366:3 (libmaster_proto.so+0xdf043) #19 kudu::master::PersistentTabletInfo::~PersistentTabletInfo() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.h:123:8 (libmaster.so+0x1610a9) #20 std::__1::default_delete::operator()(kudu::master::PersistentTabletInfo*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/memory:2285:5 (libmaster.so+0x135ce6) #21 std::__1::unique_ptr >::reset(kudu::master::PersistentTabletInfo*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/memory:2598 (libmaster.so+0x135ce6) #22 kudu::CowObject::CommitMutation() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/cow_object.h:93 (libmaster.so+0x135ce6) #23 kudu::CowGroupLock, std::__1::allocator >, kudu::master::PersistentTabletInfo>::Commit() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/cow_object.h:386:17 (libmaster.so+0x138081) #24 kudu::master::CatalogManager::ProcessTabletReport(kudu::master::TSDescriptor*, kudu::master::TabletReportPB const&, kudu::master::TabletReportUpdatesPB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:4277:16 (libmaster.so+0x126419) #25 kudu::master::MasterServiceImpl::TSHeartbeat(kudu::master::TSHeartbeatRequestPB const*, kudu::master::TSHeartbeatResponsePB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/master_service.cc:259:44 (libmaster.so+0x19e650) #26 kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_1::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/master/master.service.cc:151:13 (libmaster_proto.so+0x165854) #27 _ZNSt3__18__invokeIRZN4kudu6master15MasterServiceIfC1ERK13scoped_refptrINS1_12MetricEntityEERKS4_INS1_3rpc13ResultTrackerEEE3$_1JPKN6google8protobuf7MessageEPSI_PNS9_10RpcContextEEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSO_DpOSP_ /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/type_traits:4482:1 (libmaster_proto.so+0x1657e1) #28 void std::__1::__invoke_void_return_wrapper::__call const&, scoped_refptr const&)::$_1&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*>(kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_1&, google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/__functional_base:349 (libmaster_proto.so+0x1657e1) #29 std::__1::__function::__func const&, scoped_refptr const&)::$_1, std::__1::allocator const&, scoped_refptr const&)::$_1>, void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1562:12 (libmaster_proto.so+0x1656f4) #30 std::__1::function::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1916:12 (libkrpc.so+0xf2f29) #31 kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_if.cc:139:3 (libkrpc.so+0xf29e5) #32 kudu::rpc::ServicePool::RunThread() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_pool.cc:225:15 (libkrpc.so+0xf3dcc) #33 boost::_mfi::mf0::operator()(kudu::rpc::ServicePool*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xf5af9) I0918 20:39:43.083726 28308 raft_consensus.cc:2350] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d32ba85bbbe247d7945b09267f723959 in term 3. #34 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xf5a4a) #35 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xf59d3) #36 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xf57c9) #37 boost::function0::operator()() const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xba0b1) #38 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.cc:657:3 (libkudu_util.so+0x1ee174) Previous read of size 8 at 0x7b0c0022e880 by thread T433 (mutexes: read M694252858918829320): #0 strings::memeq(void const*, void const*, unsigned long) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/gutil/strings/fastmem.h:62:42 (libitest_util.so+0x5ec27) #1 operator==(StringPiece, StringPiece) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/gutil/strings/stringpiece.h:314:7 (libitest_util.so+0x5eaa1) #2 std::__1::equal_to::operator()(StringPiece const&, StringPiece const&) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:670:21 (libmaster.so+0x159867) #3 google::dense_hashtable, StringPiece, GoodFastHash, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SelectKey, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SetKey, std::__1::equal_to, google::libc_allocator_with_realloc > >::KeyInfo::equals(StringPiece const&, StringPiece const&) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/common/include/sparsehash/internal/densehashtable.h:1287 (libmaster.so+0x159867) #4 google::dense_hashtable, StringPiece, GoodFastHash, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SelectKey, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SetKey, std::__1::equal_to, google::libc_allocator_with_realloc > >::equals(StringPiece const&, StringPiece const&) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/common/include/sparsehash/internal/densehashtable.h:1299:21 (libmaster.so+0x1596dd) #5 google::dense_hashtable, StringPiece, GoodFastHash, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SelectKey, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SetKey, std::__1::equal_to, google::libc_allocator_with_realloc > >::find_position(StringPiece const&) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/common/include/sparsehash/internal/densehashtable.h:879:18 (libmaster.so+0x171b9b) #6 google::dense_hashtable, StringPiece, GoodFastHash, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SelectKey, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SetKey, std::__1::equal_to, google::libc_allocator_with_realloc > >::find(StringPiece const&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/common/include/sparsehash/internal/densehashtable.h:892:43 (libmaster.so+0x171974) #7 google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::find(StringPiece const&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/common/include/sparsehash/dense_hash_map:249:51 (libmaster.so+0x171809) #8 std::__1::pair, std::__1::equal_to, google::libc_allocator_with_realloc > >::mapped_type* const, bool> ComputeIfAbsentReturnAbsense, std::__1::equal_to, google::libc_allocator_with_realloc > >, kudu::master::CatalogManager::BuildLocationsForTablet(scoped_refptr const&, kudu::master::ReplicaTypeFilter, kudu::master::TabletLocationsPB*, kudu::master::CatalogManager::TSInfosDict*)::$_17>(google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >*, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::key_type const&, kudu::master::CatalogManager::BuildLocationsForTablet(scoped_refptr const&, kudu::master::ReplicaTypeFilter, kudu::master::TabletLocationsPB*, kudu::master::CatalogManager::TSInfosDict*)::$_17) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/gutil/map-util.h:883:53 (libmaster.so+0x13048a) #9 google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::mapped_type* const ComputeIfAbsent, std::__1::equal_to, google::libc_allocator_with_realloc > >, kudu::master::CatalogManager::BuildLocationsForTablet(scoped_refptr const&, kudu::master::ReplicaTypeFilter, kudu::master::TabletLocationsPB*, kudu::master::CatalogManager::TSInfosDict*)::$_17>(google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >*, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::key_type const&, kudu::master::CatalogManager::BuildLocationsForTablet(scoped_refptr const&, kudu::master::ReplicaTypeFilter, kudu::master::TabletLocationsPB*, kudu::master::CatalogManager::TSInfosDict*)::$_17) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/gutil/map-util.h:906:10 (libmaster.so+0x12a4c0) #10 kudu::master::CatalogManager::BuildLocationsForTablet(scoped_refptr const&, kudu::master::ReplicaTypeFilter, kudu::master::TabletLocationsPB*, kudu::master::CatalogManager::TSInfosDict*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:4846:18 (libmaster.so+0x12a075) #11 kudu::master::CatalogManager::GetTableLocations(kudu::master::GetTableLocationsRequestPB const*, kudu::master::GetTableLocationsResponsePB*, boost::optional, std::__1::allocator > const&>) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:5028:16 (libmaster.so+0x12b888) #12 kudu::master::MasterServiceImpl::GetTableLocations(kudu::master::GetTableLocationsRequestPB const*, kudu::master::GetTableLocationsResponsePB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/master_service.cc:441:42 (libmaster.so+0x19fcf5) #13 kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_21::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/master/master.service.cc:351:13 (libmaster_proto.so+0x169f97) I0918 20:39:43.085634 28186 leader_election.cc:310] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a68e2c30ad9f4733bbf8eb54594ce86c, d32ba85bbbe247d7945b09267f723959; no voters: 8654236fd4b14655929a78c02e6e8021 I0918 20:39:43.086946 28479 raft_consensus.cc:2644] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 3 FOLLOWER]: Leader election won for term 3 #14 _ZNSt3__18__invokeIRZN4kudu6master15MasterServiceIfC1ERK13scoped_refptrINS1_12MetricEntityEERKS4_INS1_3rpc13ResultTrackerEEE4$_21JPKN6google8protobuf7MessageEPSI_PNS9_10RpcContextEEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSO_DpOSP_ /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/type_traits:4482:1 (libmaster_proto.so+0x169f21) #15 void std::__1::__invoke_void_return_wrapper::__call const&, scoped_refptr const&)::$_21&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*>(kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_21&, google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/__functional_base:349 (libmaster_proto.so+0x169f21) #16 std::__1::__function::__func const&, scoped_refptr const&)::$_21, std::__1::allocator const&, scoped_refptr const&)::$_21>, void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1562:12 (libmaster_proto.so+0x169e34) #17 std::__1::function::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1916:12 (libkrpc.so+0xf2f29) #18 kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_if.cc:139:3 (libkrpc.so+0xf29e5) #19 kudu::rpc::ServicePool::RunThread() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_pool.cc:225:15 (libkrpc.so+0xf3dcc) #20 boost::_mfi::mf0::operator()(kudu::rpc::ServicePool*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xf5af9) #21 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xf5a4a) #22 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xf59d3) #23 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xf57c9) #24 boost::function0::operator()() const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xba0b1) #25 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.cc:657:3 (libkudu_util.so+0x1ee174) Mutex M694252858918829320 is already destroyed. Mutex M607000 (0x7b6800108028) created at: #0 pthread_mutex_init /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1211 (ksck_remote-test+0x45dcb5) #1 kudu::Mutex::Mutex() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/mutex.cc:65:8 (libkudu_util.so+0x1b5f6f) #2 kudu::RWCLock::RWCLock() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/rwc_lock.cc:30:10 (libkudu_util.so+0x1dd16d) #3 kudu::CowObject::CowObject() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/cow_object.h:40:3 (libmaster.so+0x14139a) #4 kudu::master::TabletInfo::TabletInfo(scoped_refptr, std::__1::basic_string, std::__1::allocator >) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:5302:13 (libmaster.so+0x12d995) #5 kudu::master::CatalogManager::CreateTabletInfo(scoped_refptr const&, kudu::PartitionPB const&, boost::optional, std::__1::allocator > > const&) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:1822:40 (libmaster.so+0x111ae4) #6 kudu::master::CatalogManager::CreateTable(kudu::master::CreateTableRequestPB const*, kudu::master::CreateTableResponsePB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:1676:26 (libmaster.so+0x10f90b) #7 kudu::master::MasterServiceImpl::CreateTable(kudu::master::CreateTableRequestPB const*, kudu::master::CreateTableResponsePB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/master_service.cc:341:42 (libmaster.so+0x19f1ef) #8 kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_7::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/master/master.service.cc:211:13 (libmaster_proto.so+0x166db4) #9 _ZNSt3__18__invokeIRZN4kudu6master15MasterServiceIfC1ERK13scoped_refptrINS1_12MetricEntityEERKS4_INS1_3rpc13ResultTrackerEEE3$_7JPKN6google8protobuf7MessageEPSI_PNS9_10RpcContextEEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSO_DpOSP_ /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/type_traits:4482:1 (libmaster_proto.so+0x166d41) #10 void std::__1::__invoke_void_return_wrapper::__call const&, scoped_refptr const&)::$_7&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*>(kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_7&, google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/__functional_base:349 (libmaster_proto.so+0x166d41) #11 std::__1::__function::__func const&, scoped_refptr const&)::$_7, std::__1::allocator const&, scoped_refptr const&)::$_7>, void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1562:12 (libmaster_proto.so+0x166c54) #12 std::__1::function::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1916:12 (libkrpc.so+0xf2f29) #13 kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_if.cc:139:3 (libkrpc.so+0xf29e5) #14 kudu::rpc::ServicePool::RunThread() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_pool.cc:225:15 (libkrpc.so+0xf3dcc) #15 boost::_mfi::mf0::operator()(kudu::rpc::ServicePool*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xf5af9) #16 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xf5a4a) #17 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xf59d3) #18 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xf57c9) #19 boost::function0::operator()() const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xba0b1) #20 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.cc:657:3 (libkudu_util.so+0x1ee174) Thread T8 'rpc worker-2807' (tid=28078, running) created by main thread at: #0 pthread_create /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (ksck_remote-test+0x44b806) #1 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.cc:601:15 (libkudu_util.so+0x1ed95b) #2 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::rpc::ServicePool::* const&)(), kudu::rpc::ServicePool* const&, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.h:164:12 (libkrpc.so+0xf4ee5) #3 kudu::rpc::ServicePool::Init(int) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_pool.cc:92:5 (libkrpc.so+0xf39d6) #4 kudu::RpcServer::RegisterService(gscoped_ptr >) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/server/rpc_server.cc:157:3 (libserver_process.so+0x83529) #5 kudu::server::ServerBase::RegisterService(gscoped_ptr >) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/server/server_base.cc:638:23 (libserver_process.so+0x8a0ff) #6 kudu::master::Master::StartAsync() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/master.cc:186:3 (libmaster.so+0x18830f) #7 kudu::master::MiniMaster::Start() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/mini_master.cc:94:3 (libmaster.so+0x1a7e64) #8 kudu::cluster::InternalMiniCluster::StartMasters() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/mini-cluster/internal_mini_cluster.cc:160:5 (libmini_cluster.so+0x42bf3) #9 kudu::cluster::InternalMiniCluster::Start() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/mini-cluster/internal_mini_cluster.cc:99:3 (libmini_cluster.so+0x426cd) #10 kudu::tools::RemoteKsckTest::SetUp() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/tools/ksck_remote-test.cc:123:5 (ksck_remote-test+0x4ef737) #11 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef) #12 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef) #13 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2470:3 (libgmock.so+0x343c1) #14 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c) #15 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226) #16 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa) #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f) #18 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f) #19 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2) #20 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x351b) #21 main /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2cc6) Thread T433 'rpc worker-2808' (tid=28080, running) created by main thread at: #0 pthread_create /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (ksck_remote-test+0x44b806) #1 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.cc:601:15 (libkudu_util.so+0x1ed95b) #2 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::rpc::ServicePool::* const&)(), kudu::rpc::ServicePool* const&, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.h:164:12 (libkrpc.so+0xf4ee5) #3 kudu::rpc::ServicePool::Init(int) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_pool.cc:92:5 (libkrpc.so+0xf39d6) #4 kudu::RpcServer::RegisterService(gscoped_ptr >) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/server/rpc_server.cc:157:3 (libserver_process.so+0x83529) #5 kudu::server::ServerBase::RegisterService(gscoped_ptr >) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/server/server_base.cc:638:23 (libserver_process.so+0x8a0ff) #6 kudu::master::Master::StartAsync() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/master.cc:186:3 (libmaster.so+0x18830f) #7 kudu::master::MiniMaster::Start() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/mini_master.cc:94:3 (libmaster.so+0x1a7e64) #8 kudu::cluster::InternalMiniCluster::StartMasters() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/mini-cluster/internal_mini_cluster.cc:160:5 (libmini_cluster.so+0x42bf3) #9 kudu::cluster::InternalMiniCluster::Start() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/mini-cluster/internal_mini_cluster.cc:99:3 (libmini_cluster.so+0x426cd) #10 kudu::tools::RemoteKsckTest::SetUp() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/tools/ksck_remote-test.cc:123:5 (ksck_remote-test+0x4ef737) #11 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef) #12 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef) #13 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2470:3 (libgmock.so+0x343c1) #14 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c) #15 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226) #16 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa) #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f) #18 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f) #19 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2) #20 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x351b) #21 main /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2cc6) SUMMARY: ThreadSanitizer: data race /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 in operator delete(void*) ================== I0918 20:39:43.105653 28479 raft_consensus.cc:667] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 3 LEADER]: Becoming Leader. State: Replica: d32ba85bbbe247d7945b09267f723959, State: Running, Role: LEADER I0918 20:39:43.107115 28479 consensus_queue.cc:206] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } I0918 20:39:43.110790 28308 raft_consensus.cc:1149] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 3 FOLLOWER]: Rejecting Update request from peer 8654236fd4b14655929a78c02e6e8021 for earlier term 2. Current term is 3. Ops: [] I0918 20:39:43.114902 28495 consensus_queue.cc:995] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s I0918 20:39:43.116173 28495 raft_consensus.cc:2885] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 2 LEADER]: Stepping down as leader of term 2 I0918 20:39:43.116462 28495 raft_consensus.cc:705] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8654236fd4b14655929a78c02e6e8021, State: Running, Role: LEADER I0918 20:39:43.118178 28495 consensus_queue.cc:229] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } } I0918 20:39:43.119428 28495 raft_consensus.cc:2890] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 2 FOLLOWER]: Advancing to term 3 ================== WARNING: ThreadSanitizer: data race (pid=19320) Write of size 8 at 0x7b0c0022e888 by thread T8 (mutexes: read M694252858918829320, write M607000): #0 operator delete(void*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 (ksck_remote-test+0x4db331) #1 __libcpp_deallocate /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/libcxx/include/new:236:3 (libc++.so.1+0xd7ab0) #2 deallocate /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/libcxx/include/memory:1796 (libc++.so.1+0xd7ab0) #3 deallocate /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/libcxx/include/memory:1555 (libc++.so.1+0xd7ab0) #4 std::__1::basic_string, std::__1::allocator >::~basic_string() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/libcxx/include/string:1941 (libc++.so.1+0xd7ab0) #5 google::protobuf::internal::ArenaStringPtr::DestroyNoArena(std::__1::basic_string, std::__1::allocator > const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/google/protobuf/arenastring.h:258:7 (libclient_proto.so+0x16e02) #6 kudu::consensus::RaftPeerPB::SharedDtor() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:1039:19 (libconsensus_metadata_proto.so+0x18a63) #7 kudu::consensus::RaftPeerPB::~RaftPeerPB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:1035:3 (libconsensus_metadata_proto.so+0x18a03) #8 kudu::consensus::RaftPeerPB::~RaftPeerPB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:1033:27 (libconsensus_metadata_proto.so+0x18b29) #9 google::protobuf::internal::GenericTypeHandler::Delete(kudu::consensus::RaftPeerPB*, google::protobuf::Arena*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/google/protobuf/repeated_field.h:615:7 (libconsensus.so+0xc0920) #10 void google::protobuf::internal::RepeatedPtrFieldBase::Destroy::TypeHandler>() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/google/protobuf/repeated_field.h:1429:7 (libconsensus.so+0xc08a9) #11 google::protobuf::RepeatedPtrField::~RepeatedPtrField() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/google/protobuf/repeated_field.h:1892:3 (libconsensus.so+0xbfaaa) #12 kudu::consensus::RaftConfigPB::~RaftConfigPB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:1699:1 (libconsensus_metadata_proto.so+0x19ea4) #13 kudu::consensus::RaftConfigPB::~RaftConfigPB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:1696:31 (libconsensus_metadata_proto.so+0x19ef9) #14 kudu::consensus::ConsensusStatePB::SharedDtor() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:2189:44 (libconsensus_metadata_proto.so+0x1afb0) #15 kudu::consensus::ConsensusStatePB::~ConsensusStatePB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:2184:3 (libconsensus_metadata_proto.so+0x1af23) #16 kudu::consensus::ConsensusStatePB::~ConsensusStatePB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/consensus/metadata.pb.cc:2182:39 (libconsensus_metadata_proto.so+0x1b019) #17 kudu::master::SysTabletsEntryPB::SharedDtor() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/master/master.pb.cc:3375:44 (libmaster_proto.so+0xdf114) #18 kudu::master::SysTabletsEntryPB::~SysTabletsEntryPB() /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/master/master.pb.cc:3366:3 (libmaster_proto.so+0xdf043) #19 kudu::master::PersistentTabletInfo::~PersistentTabletInfo() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.h:123:8 (libmaster.so+0x1610a9) #20 std::__1::default_delete::operator()(kudu::master::PersistentTabletInfo*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/memory:2285:5 (libmaster.so+0x135ce6) #21 std::__1::unique_ptr >::reset(kudu::master::PersistentTabletInfo*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/memory:2598 (libmaster.so+0x135ce6) #22 kudu::CowObject::CommitMutation() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/cow_object.h:93 (libmaster.so+0x135ce6) #23 kudu::CowGroupLock, std::__1::allocator >, kudu::master::PersistentTabletInfo>::Commit() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/cow_object.h:386:17 (libmaster.so+0x138081) #24 kudu::master::CatalogManager::ProcessTabletReport(kudu::master::TSDescriptor*, kudu::master::TabletReportPB const&, kudu::master::TabletReportUpdatesPB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:4277:16 (libmaster.so+0x126419) #25 kudu::master::MasterServiceImpl::TSHeartbeat(kudu::master::TSHeartbeatRequestPB const*, kudu::master::TSHeartbeatResponsePB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/master_service.cc:259:44 (libmaster.so+0x19e650) #26 kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_1::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/master/master.service.cc:151:13 (libmaster_proto.so+0x165854) #27 _ZNSt3__18__invokeIRZN4kudu6master15MasterServiceIfC1ERK13scoped_refptrINS1_12MetricEntityEERKS4_INS1_3rpc13ResultTrackerEEE3$_1JPKN6google8protobuf7MessageEPSI_PNS9_10RpcContextEEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSO_DpOSP_ /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/type_traits:4482:1 (libmaster_proto.so+0x1657e1) #28 void std::__1::__invoke_void_return_wrapper::__call const&, scoped_refptr const&)::$_1&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*>(kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_1&, google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/__functional_base:349 (libmaster_proto.so+0x1657e1) #29 std::__1::__function::__func const&, scoped_refptr const&)::$_1, std::__1::allocator const&, scoped_refptr const&)::$_1>, void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1562:12 (libmaster_proto.so+0x1656f4) #30 std::__1::function::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1916:12 (libkrpc.so+0xf2f29) #31 kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_if.cc:139:3 (libkrpc.so+0xf29e5) #32 kudu::rpc::ServicePool::RunThread() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_pool.cc:225:15 (libkrpc.so+0xf3dcc) #33 boost::_mfi::mf0::operator()(kudu::rpc::ServicePool*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xf5af9) #34 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xf5a4a) #35 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xf59d3) #36 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xf57c9) #37 boost::function0::operator()() const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xba0b1) #38 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.cc:657:3 (libkudu_util.so+0x1ee174) Previous read of size 8 at 0x7b0c0022e888 by thread T433 (mutexes: read M694252858918829320): #0 strings::memeq(void const*, void const*, unsigned long) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/gutil/strings/fastmem.h:47:44 (libitest_util.so+0x5eb8c) #1 operator==(StringPiece, StringPiece) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/gutil/strings/stringpiece.h:314:7 (libitest_util.so+0x5eaa1) #2 std::__1::equal_to::operator()(StringPiece const&, StringPiece const&) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:670:21 (libmaster.so+0x159867) #3 google::dense_hashtable, StringPiece, GoodFastHash, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SelectKey, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SetKey, std::__1::equal_to, google::libc_allocator_with_realloc > >::KeyInfo::equals(StringPiece const&, StringPiece const&) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/common/include/sparsehash/internal/densehashtable.h:1287 (libmaster.so+0x159867) #4 google::dense_hashtable, StringPiece, GoodFastHash, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SelectKey, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SetKey, std::__1::equal_to, google::libc_allocator_with_realloc > >::equals(StringPiece const&, StringPiece const&) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/common/include/sparsehash/internal/densehashtable.h:1299:21 (libmaster.so+0x1596dd) #5 google::dense_hashtable, StringPiece, GoodFastHash, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SelectKey, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SetKey, std::__1::equal_to, google::libc_allocator_with_realloc > >::find_position(StringPiece const&) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/common/include/sparsehash/internal/densehashtable.h:879:18 (libmaster.so+0x171b9b) #6 google::dense_hashtable, StringPiece, GoodFastHash, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SelectKey, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::SetKey, std::__1::equal_to, google::libc_allocator_with_realloc > >::find(StringPiece const&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/common/include/sparsehash/internal/densehashtable.h:892:43 (libmaster.so+0x171974) #7 google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::find(StringPiece const&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/common/include/sparsehash/dense_hash_map:249:51 (libmaster.so+0x171809) #8 std::__1::pair, std::__1::equal_to, google::libc_allocator_with_realloc > >::mapped_type* const, bool> ComputeIfAbsentReturnAbsense, std::__1::equal_to, google::libc_allocator_with_realloc > >, kudu::master::CatalogManager::BuildLocationsForTablet(scoped_refptr const&, kudu::master::ReplicaTypeFilter, kudu::master::TabletLocationsPB*, kudu::master::CatalogManager::TSInfosDict*)::$_17>(google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >*, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::key_type const&, kudu::master::CatalogManager::BuildLocationsForTablet(scoped_refptr const&, kudu::master::ReplicaTypeFilter, kudu::master::TabletLocationsPB*, kudu::master::CatalogManager::TSInfosDict*)::$_17) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/gutil/map-util.h:883:53 (libmaster.so+0x13048a) #9 google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::mapped_type* const ComputeIfAbsent, std::__1::equal_to, google::libc_allocator_with_realloc > >, kudu::master::CatalogManager::BuildLocationsForTablet(scoped_refptr const&, kudu::master::ReplicaTypeFilter, kudu::master::TabletLocationsPB*, kudu::master::CatalogManager::TSInfosDict*)::$_17>(google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >*, google::dense_hash_map, std::__1::equal_to, google::libc_allocator_with_realloc > >::key_type const&, kudu::master::CatalogManager::BuildLocationsForTablet(scoped_refptr const&, kudu::master::ReplicaTypeFilter, kudu::master::TabletLocationsPB*, kudu::master::CatalogManager::TSInfosDict*)::$_17) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/gutil/map-util.h:906:10 (libmaster.so+0x12a4c0) #10 kudu::master::CatalogManager::BuildLocationsForTablet(scoped_refptr const&, kudu::master::ReplicaTypeFilter, kudu::master::TabletLocationsPB*, kudu::master::CatalogManager::TSInfosDict*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:4846:18 (libmaster.so+0x12a075) #11 kudu::master::CatalogManager::GetTableLocations(kudu::master::GetTableLocationsRequestPB const*, kudu::master::GetTableLocationsResponsePB*, boost::optional, std::__1::allocator > const&>) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:5028:16 (libmaster.so+0x12b888) #12 kudu::master::MasterServiceImpl::GetTableLocations(kudu::master::GetTableLocationsRequestPB const*, kudu::master::GetTableLocationsResponsePB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/master_service.cc:441:42 (libmaster.so+0x19fcf5) #13 kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_21::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/master/master.service.cc:351:13 (libmaster_proto.so+0x169f97) #14 _ZNSt3__18__invokeIRZN4kudu6master15MasterServiceIfC1ERK13scoped_refptrINS1_12MetricEntityEERKS4_INS1_3rpc13ResultTrackerEEE4$_21JPKN6google8protobuf7MessageEPSI_PNS9_10RpcContextEEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSO_DpOSP_ /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/type_traits:4482:1 (libmaster_proto.so+0x169f21) #15 void std::__1::__invoke_void_return_wrapper::__call const&, scoped_refptr const&)::$_21&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*>(kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_21&, google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/__functional_base:349 (libmaster_proto.so+0x169f21) #16 std::__1::__function::__func const&, scoped_refptr const&)::$_21, std::__1::allocator const&, scoped_refptr const&)::$_21>, void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1562:12 (libmaster_proto.so+0x169e34) #17 std::__1::function::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1916:12 (libkrpc.so+0xf2f29) #18 kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_if.cc:139:3 (libkrpc.so+0xf29e5) #19 kudu::rpc::ServicePool::RunThread() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_pool.cc:225:15 (libkrpc.so+0xf3dcc) #20 boost::_mfi::mf0::operator()(kudu::rpc::ServicePool*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xf5af9) #21 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xf5a4a) #22 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xf59d3) #23 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xf57c9) #24 boost::function0::operator()() const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xba0b1) #25 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.cc:657:3 (libkudu_util.so+0x1ee174) Mutex M694252858918829320 is already destroyed. Mutex M607000 (0x7b6800108028) created at: #0 pthread_mutex_init /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1211 (ksck_remote-test+0x45dcb5) #1 kudu::Mutex::Mutex() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/mutex.cc:65:8 (libkudu_util.so+0x1b5f6f) #2 kudu::RWCLock::RWCLock() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/rwc_lock.cc:30:10 (libkudu_util.so+0x1dd16d) #3 kudu::CowObject::CowObject() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/cow_object.h:40:3 (libmaster.so+0x14139a) #4 kudu::master::TabletInfo::TabletInfo(scoped_refptr, std::__1::basic_string, std::__1::allocator >) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:5302:13 (libmaster.so+0x12d995) #5 kudu::master::CatalogManager::CreateTabletInfo(scoped_refptr const&, kudu::PartitionPB const&, boost::optional, std::__1::allocator > > const&) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:1822:40 (libmaster.so+0x111ae4) #6 kudu::master::CatalogManager::CreateTable(kudu::master::CreateTableRequestPB const*, kudu::master::CreateTableResponsePB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/catalog_manager.cc:1676:26 (libmaster.so+0x10f90b) #7 kudu::master::MasterServiceImpl::CreateTable(kudu::master::CreateTableRequestPB const*, kudu::master::CreateTableResponsePB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/master_service.cc:341:42 (libmaster.so+0x19f1ef) #8 kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_7::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/build/tsan/src/kudu/master/master.service.cc:211:13 (libmaster_proto.so+0x166db4) #9 _ZNSt3__18__invokeIRZN4kudu6master15MasterServiceIfC1ERK13scoped_refptrINS1_12MetricEntityEERKS4_INS1_3rpc13ResultTrackerEEE3$_7JPKN6google8protobuf7MessageEPSI_PNS9_10RpcContextEEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSO_DpOSP_ /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/type_traits:4482:1 (libmaster_proto.so+0x166d41) #10 void std::__1::__invoke_void_return_wrapper::__call const&, scoped_refptr const&)::$_7&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*>(kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr const&, scoped_refptr const&)::$_7&, google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/__functional_base:349 (libmaster_proto.so+0x166d41) #11 std::__1::__function::__func const&, scoped_refptr const&)::$_7, std::__1::allocator const&, scoped_refptr const&)::$_7>, void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1562:12 (libmaster_proto.so+0x166c54) #12 std::__1::function::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/c++/v1/functional:1916:12 (libkrpc.so+0xf2f29) #13 kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_if.cc:139:3 (libkrpc.so+0xf29e5) #14 kudu::rpc::ServicePool::RunThread() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_pool.cc:225:15 (libkrpc.so+0xf3dcc) #15 boost::_mfi::mf0::operator()(kudu::rpc::ServicePool*) const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xf5af9) #16 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xf5a4a) #17 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xf59d3) #18 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xf57c9) #19 boost::function0::operator()() const /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xba0b1) #20 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.cc:657:3 (libkudu_util.so+0x1ee174) Thread T8 'rpc worker-2807' (tid=28078, running) created by main thread at: #0 pthread_create /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (ksck_remote-test+0x44b806) #1 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.cc:601:15 (libkudu_util.so+0x1ed95b) #2 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::rpc::ServicePool::* const&)(), kudu::rpc::ServicePool* const&, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.h:164:12 (libkrpc.so+0xf4ee5) #3 kudu::rpc::ServicePool::Init(int) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_pool.cc:92:5 (libkrpc.so+0xf39d6) #4 kudu::RpcServer::RegisterService(gscoped_ptr >) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/server/rpc_server.cc:157:3 (libserver_process.so+0x83529) #5 kudu::server::ServerBase::RegisterService(gscoped_ptr >) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/server/server_base.cc:638:23 (libserver_process.so+0x8a0ff) #6 kudu::master::Master::StartAsync() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/master.cc:186:3 (libmaster.so+0x18830f) #7 kudu::master::MiniMaster::Start() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/mini_master.cc:94:3 (libmaster.so+0x1a7e64) #8 kudu::cluster::InternalMiniCluster::StartMasters() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/mini-cluster/internal_mini_cluster.cc:160:5 (libmini_cluster.so+0x42bf3) #9 kudu::cluster::InternalMiniCluster::Start() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/mini-cluster/internal_mini_cluster.cc:99:3 (libmini_cluster.so+0x426cd) #10 kudu::tools::RemoteKsckTest::SetUp() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/tools/ksck_remote-test.cc:123:5 (ksck_remote-test+0x4ef737) #11 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef) #12 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef) #13 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2470:3 (libgmock.so+0x343c1) #14 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c) #15 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226) #16 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa) #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f) #18 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f) #19 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2) #20 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x351b) #21 main /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2cc6) Thread T433 'rpc worker-2808' (tid=28080, running) created by main thread at: #0 pthread_create /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (ksck_remote-test+0x44b806) #1 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.cc:601:15 (libkudu_util.so+0x1ed95b) #2 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::rpc::ServicePool::* const&)(), kudu::rpc::ServicePool* const&, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.h:164:12 (libkrpc.so+0xf4ee5) #3 kudu::rpc::ServicePool::Init(int) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_pool.cc:92:5 (libkrpc.so+0xf39d6) #4 kudu::RpcServer::RegisterService(gscoped_ptr >) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/server/rpc_server.cc:157:3 (libserver_process.so+0x83529) #5 kudu::server::ServerBase::RegisterService(gscoped_ptr >) /home/jenkins-slave/workspace/kudu-master/1/src/kudu/server/server_base.cc:638:23 (libserver_process.so+0x8a0ff) #6 kudu::master::Master::StartAsync() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/master.cc:186:3 (libmaster.so+0x18830f) #7 kudu::master::MiniMaster::Start() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/mini_master.cc:94:3 (libmaster.so+0x1a7e64) #8 kudu::cluster::InternalMiniCluster::StartMasters() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/mini-cluster/internal_mini_cluster.cc:160:5 (libmini_cluster.so+0x42bf3) #9 kudu::cluster::InternalMiniCluster::Start() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/mini-cluster/internal_mini_cluster.cc:99:3 (libmini_cluster.so+0x426cd) #10 kudu::tools::RemoteKsckTest::SetUp() /home/jenkins-slave/workspace/kudu-master/1/src/kudu/tools/ksck_remote-test.cc:123:5 (ksck_remote-test+0x4ef737) #11 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef) #12 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef) #13 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2470:3 (libgmock.so+0x343c1) #14 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c) #15 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226) #16 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa) #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f) #18 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f) #19 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2) #20 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/1/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x351b) #21 main /home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2cc6) SUMMARY: ThreadSanitizer: data race /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 in operator delete(void*) ================== I0918 20:39:43.183796 28227 raft_consensus.cc:1149] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 3 LEADER]: Rejecting Update request from peer 8654236fd4b14655929a78c02e6e8021 for earlier term 2. Current term is 3. Ops: [] W0918 20:39:43.185178 28188 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.222.3:41335, user_credentials={real_user=slave}} blocked reactor thread for 67827.2us I0918 20:39:43.192489 28078 catalog_manager.cc:4131] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 reported cstate change: term changed from 2 to 3, leader changed from 8654236fd4b14655929a78c02e6e8021 (127.18.222.3) to d32ba85bbbe247d7945b09267f723959 (127.18.222.1). New cstate: current_term: 3 leader_uuid: "d32ba85bbbe247d7945b09267f723959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d32ba85bbbe247d7945b09267f723959" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38599 } health_report { overall_health: HEALTHY } } } I0918 20:39:43.544564 28377 raft_consensus.cc:1184] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 3 FOLLOWER]: Refusing update from remote peer d32ba85bbbe247d7945b09267f723959: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 2. (term mismatch) I0918 20:39:43.544970 28377 pending_rounds.cc:78] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021: Aborting all transactions after (but not including) 1 I0918 20:39:43.545188 28377 pending_rounds.cc:99] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021: Aborting uncommitted NO_OP operation due to leader change: 2.2 I0918 20:39:43.545447 28377 raft_consensus.cc:2727] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader I0918 20:39:43.545717 28377 pending_rounds.cc:99] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021: Aborting uncommitted WRITE_OP operation due to leader change: 2.3 W0918 20:39:43.546339 28377 tablet_service.cc:652] failed transaction from {username='slave'} at 127.0.0.1:35062: Aborted: Transaction aborted by new leader I0918 20:39:43.549039 28508 consensus_queue.cc:984] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8654236fd4b14655929a78c02e6e8021" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 41335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:43.575000 28308 raft_consensus.cc:1184] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 3 FOLLOWER]: Refusing update from remote peer d32ba85bbbe247d7945b09267f723959: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch) I0918 20:39:43.576792 28508 consensus_queue.cc:984] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a68e2c30ad9f4733bbf8eb54594ce86c" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 46789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0918 20:39:44.209223 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:44.285481 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:44.285931 19320 tablet_replica.cc:273] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959: stopping tablet replica I0918 20:39:44.286581 19320 raft_consensus.cc:2147] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 3 LEADER]: Raft consensus shutting down. I0918 20:39:44.287452 19320 raft_consensus.cc:2174] T 0e7777829e0c46a3a5dc016b95c26a5b P d32ba85bbbe247d7945b09267f723959 [term 3 FOLLOWER]: Raft consensus is shut down! I0918 20:39:44.290475 19320 tablet_replica.cc:273] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959: stopping tablet replica I0918 20:39:44.290947 19320 raft_consensus.cc:2147] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:44.291782 19320 raft_consensus.cc:2174] T 20884cc5471a4921b39674fa4462064c P d32ba85bbbe247d7945b09267f723959 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:44.294600 19320 tablet_replica.cc:273] T 7198354ae5254f8c92805487d4542a83 P d32ba85bbbe247d7945b09267f723959: stopping tablet replica I0918 20:39:44.295104 19320 raft_consensus.cc:2147] T 7198354ae5254f8c92805487d4542a83 P d32ba85bbbe247d7945b09267f723959 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:44.295534 19320 raft_consensus.cc:2174] T 7198354ae5254f8c92805487d4542a83 P d32ba85bbbe247d7945b09267f723959 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:44.316478 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:44.316694 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:44.317114 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:44.335403 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:44.391408 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:44.391831 19320 tablet_replica.cc:273] T 20884cc5471a4921b39674fa4462064c P a68e2c30ad9f4733bbf8eb54594ce86c: stopping tablet replica I0918 20:39:44.392241 19320 raft_consensus.cc:2147] T 20884cc5471a4921b39674fa4462064c P a68e2c30ad9f4733bbf8eb54594ce86c [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:44.392546 19320 raft_consensus.cc:2174] T 20884cc5471a4921b39674fa4462064c P a68e2c30ad9f4733bbf8eb54594ce86c [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:44.394866 19320 tablet_replica.cc:273] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c: stopping tablet replica I0918 20:39:44.395314 19320 raft_consensus.cc:2147] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 3 FOLLOWER]: Raft consensus shutting down. I0918 20:39:44.395651 19320 raft_consensus.cc:2174] T 0e7777829e0c46a3a5dc016b95c26a5b P a68e2c30ad9f4733bbf8eb54594ce86c [term 3 FOLLOWER]: Raft consensus is shut down! I0918 20:39:44.398721 19320 tablet_replica.cc:273] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c: stopping tablet replica I0918 20:39:44.399098 19320 raft_consensus.cc:2147] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:44.399837 19320 raft_consensus.cc:2174] T 7198354ae5254f8c92805487d4542a83 P a68e2c30ad9f4733bbf8eb54594ce86c [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:44.425482 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:44.425695 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:44.426028 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:44.441151 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:44.487274 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:44.487610 19320 tablet_replica.cc:273] T 7198354ae5254f8c92805487d4542a83 P 8654236fd4b14655929a78c02e6e8021: stopping tablet replica I0918 20:39:44.488271 19320 raft_consensus.cc:2147] T 7198354ae5254f8c92805487d4542a83 P 8654236fd4b14655929a78c02e6e8021 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:44.488698 19320 raft_consensus.cc:2174] T 7198354ae5254f8c92805487d4542a83 P 8654236fd4b14655929a78c02e6e8021 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:44.491353 19320 tablet_replica.cc:273] T 20884cc5471a4921b39674fa4462064c P 8654236fd4b14655929a78c02e6e8021: stopping tablet replica I0918 20:39:44.491896 19320 raft_consensus.cc:2147] T 20884cc5471a4921b39674fa4462064c P 8654236fd4b14655929a78c02e6e8021 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:44.492208 19320 raft_consensus.cc:2174] T 20884cc5471a4921b39674fa4462064c P 8654236fd4b14655929a78c02e6e8021 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:44.494330 19320 tablet_replica.cc:273] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021: stopping tablet replica I0918 20:39:44.494670 19320 raft_consensus.cc:2147] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 3 FOLLOWER]: Raft consensus shutting down. I0918 20:39:44.494953 19320 raft_consensus.cc:2174] T 0e7777829e0c46a3a5dc016b95c26a5b P 8654236fd4b14655929a78c02e6e8021 [term 3 FOLLOWER]: Raft consensus is shut down! I0918 20:39:44.518304 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:44.518504 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:44.518859 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:44.532125 19320 master.cc:247] Master@127.18.222.62:37137 shutting down... I0918 20:39:44.554358 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:44.554754 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:44.554986 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 36211758fed84e26a80a3f1cd3c72444: stopping tablet replica I0918 20:39:44.573915 19320 master.cc:258] Master@127.18.222.62:37137 shutdown complete. I0918 20:39:44.586021 19320 master.cc:247] Master@127.18.222.61:38767 shutting down... I0918 20:39:44.606420 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:44.607201 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:44.607440 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P e7d0a1df3aa2499bb006d06bfbc57531: stopping tablet replica I0918 20:39:44.627440 19320 master.cc:258] Master@127.18.222.61:38767 shutdown complete. I0918 20:39:44.640836 19320 master.cc:247] Master@127.18.222.60:44875 shutting down... I0918 20:39:44.663583 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:44.664055 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:44.664305 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 80e9151b52aa4ba6bed4e6336b24a35e: stopping tablet replica I0918 20:39:44.681974 19320 master.cc:258] Master@127.18.222.60:44875 shutdown complete. [ OK ] RemoteKsckTest.TestMatchingTabletIdFilter (5937 ms) [ RUN ] RemoteKsckTest.TestTableFiltersNoMatch I0918 20:39:44.717605 19320 test_util.cc:212] Using random seed: -699361525 I0918 20:39:44.718380 19320 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.18.222.62:40573,127.18.222.61:40165,127.18.222.60:42325 I0918 20:39:44.731281 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 169592us I0918 20:39:44.731650 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:44.732095 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:44.732306 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:44.732421 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:44.733506 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/master-0-root/instance: uuid: "02d4a9e1c4ac49fbbd722def96184901" format_stamp: "Formatted at 2019-09-18 20:39:44 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:44.742236 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s I0918 20:39:44.743844 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:44.750551 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:44.750738 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/master-0-root uuid: "02d4a9e1c4ac49fbbd722def96184901" format_stamp: "Formatted at 2019-09-18 20:39:44 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:44.750969 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:44.823307 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:40573 I0918 20:39:44.825120 19320 webserver.cc:374] Webserver started at http://127.18.222.62:39281/ using document root and password file I0918 20:39:44.826915 28603 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:44.831866 28603 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:44.841759 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 169592us I0918 20:39:44.842145 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:44.842325 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:44.842564 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:44.842686 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:44.844079 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/master-1-root/instance: uuid: "6fc25e406a11476584aea076bc391255" format_stamp: "Formatted at 2019-09-18 20:39:44 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:44.846050 28603 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:44.847918 28603 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:40165: Network error: Client connection negotiation failed: client connection to 127.18.222.61:40165: connect: Connection refused (error 111) I0918 20:39:44.854344 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.006s sys 0.004s I0918 20:39:44.856048 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:44.863759 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:44.863993 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/master-1-root uuid: "6fc25e406a11476584aea076bc391255" format_stamp: "Formatted at 2019-09-18 20:39:44 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:44.864317 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:44.867398 28603 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } attempt: 1 W0918 20:39:44.869612 28603 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:40165: Network error: Client connection negotiation failed: client connection to 127.18.222.61:40165: connect: Connection refused (error 111) I0918 20:39:44.939208 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:40165 I0918 20:39:44.941339 19320 webserver.cc:374] Webserver started at http://127.18.222.61:40887/ using document root and password file I0918 20:39:44.943825 28660 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:44.945029 28603 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } attempt: 2 I0918 20:39:44.952505 28660 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:44.960851 28603 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:44.963318 28603 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:42325: Network error: Client connection negotiation failed: client connection to 127.18.222.60:42325: connect: Connection refused (error 111) I0918 20:39:44.967137 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 169592us I0918 20:39:44.967643 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:44.967895 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:44.968161 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:44.968288 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:44.968921 28660 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:44.969432 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/master-2-root/instance: uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" format_stamp: "Formatted at 2019-09-18 20:39:44 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:44.980903 28660 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:44.983019 28660 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:42325: Network error: Client connection negotiation failed: client connection to 127.18.222.60:42325: connect: Connection refused (error 111) I0918 20:39:44.983927 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.014s user 0.010s sys 0.005s I0918 20:39:44.985621 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:44.993692 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:44.993890 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/master-2-root uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" format_stamp: "Formatted at 2019-09-18 20:39:44 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:44.994125 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:45.011776 28603 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } attempt: 1 W0918 20:39:45.013953 28603 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:42325: Network error: Client connection negotiation failed: client connection to 127.18.222.60:42325: connect: Connection refused (error 111) I0918 20:39:45.045567 28660 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } attempt: 1 W0918 20:39:45.048126 28660 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:42325: Network error: Client connection negotiation failed: client connection to 127.18.222.60:42325: connect: Connection refused (error 111) I0918 20:39:45.059402 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:42325 I0918 20:39:45.061354 19320 webserver.cc:374] Webserver started at http://127.18.222.60:32895/ using document root and password file I0918 20:39:45.061357 28603 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } attempt: 2 I0918 20:39:45.062528 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0918 20:39:45.064118 28717 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:45.071316 28717 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:45.082476 28603 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901: Bootstrap starting. I0918 20:39:45.086766 28603 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901: No blocks or log segments found. Creating new log. I0918 20:39:45.086885 28717 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:45.090358 28603 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901: No bootstrap required, opened a new log I0918 20:39:45.092687 28603 raft_consensus.cc:340] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } I0918 20:39:45.093240 28603 raft_consensus.cc:366] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:45.093485 28603 raft_consensus.cc:705] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d4a9e1c4ac49fbbd722def96184901, State: Initialized, Role: FOLLOWER I0918 20:39:45.094135 28603 consensus_queue.cc:229] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } I0918 20:39:45.096141 28603 sys_catalog.cc:340] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } } I0918 20:39:45.096706 28603 sys_catalog.cc:343] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:45.096493 28721 sys_catalog.cc:340] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } } I0918 20:39:45.097415 28721 sys_catalog.cc:343] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:45.098100 28603 sys_catalog.cc:428] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:45.099347 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 I0918 20:39:45.100486 28717 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:45.101867 28722 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:45.102313 28722 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:45.116278 28717 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97: Bootstrap starting. I0918 20:39:45.119259 28717 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97: No blocks or log segments found. Creating new log. I0918 20:39:45.122431 28717 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97: No bootstrap required, opened a new log I0918 20:39:45.124068 28717 raft_consensus.cc:340] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } I0918 20:39:45.124583 28717 raft_consensus.cc:366] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:45.124779 28717 raft_consensus.cc:705] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548b97f9463d4a3e9e3c60bf73bc1b97, State: Initialized, Role: FOLLOWER I0918 20:39:45.125277 28717 consensus_queue.cc:229] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } I0918 20:39:45.126576 28660 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } attempt: 2 I0918 20:39:45.126700 28717 sys_catalog.cc:340] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } } I0918 20:39:45.127462 28717 sys_catalog.cc:343] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:45.127094 28724 sys_catalog.cc:340] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } } I0918 20:39:45.127975 28724 sys_catalog.cc:343] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:45.128785 28717 sys_catalog.cc:428] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97 [sys.catalog]: configured and running, proceeding with master startup. W0918 20:39:45.133224 28725 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:45.133448 28725 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:45.143465 28660 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255: Bootstrap starting. I0918 20:39:45.146368 28660 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255: No blocks or log segments found. Creating new log. I0918 20:39:45.149574 28660 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255: No bootstrap required, opened a new log I0918 20:39:45.151106 28660 raft_consensus.cc:340] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } I0918 20:39:45.151659 28660 raft_consensus.cc:366] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:45.151890 28660 raft_consensus.cc:705] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc25e406a11476584aea076bc391255, State: Initialized, Role: FOLLOWER I0918 20:39:45.152483 28660 consensus_queue.cc:229] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } I0918 20:39:45.154146 28660 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } } I0918 20:39:45.154871 28660 sys_catalog.cc:343] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:45.154358 28727 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } } I0918 20:39:45.155215 28727 sys_catalog.cc:343] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:45.156049 28660 sys_catalog.cc:428] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:45.157222 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 W0918 20:39:45.159260 28728 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:45.159539 28728 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:45.172961 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 170092us I0918 20:39:45.173382 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:45.173557 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:45.173779 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:45.173915 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:45.174975 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-0-root/instance: uuid: "7738715631eb46b18c9c15fa8afa48cd" format_stamp: "Formatted at 2019-09-18 20:39:45 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:45.185389 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.005s sys 0.008s I0918 20:39:45.187021 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:45.195034 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:45.195300 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-0-root uuid: "7738715631eb46b18c9c15fa8afa48cd" format_stamp: "Formatted at 2019-09-18 20:39:45 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:45.195571 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:45.229166 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:45.229359 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:45.248425 28727 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:45.249049 28727 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } I0918 20:39:45.251186 28727 leader_election.cc:296] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02d4a9e1c4ac49fbbd722def96184901 (127.18.222.62:40573), 548b97f9463d4a3e9e3c60bf73bc1b97 (127.18.222.60:42325) I0918 20:39:45.253089 28580 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fc25e406a11476584aea076bc391255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d4a9e1c4ac49fbbd722def96184901" is_pre_election: true I0918 20:39:45.253712 28694 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fc25e406a11476584aea076bc391255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" is_pre_election: true I0918 20:39:45.254253 28580 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fc25e406a11476584aea076bc391255 in term 0. I0918 20:39:45.254489 28694 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fc25e406a11476584aea076bc391255 in term 0. I0918 20:39:45.256023 28611 leader_election.cc:310] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d4a9e1c4ac49fbbd722def96184901, 6fc25e406a11476584aea076bc391255; no voters: I0918 20:39:45.257633 28727 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:45.258023 28727 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:45.258417 28727 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:45.266129 28727 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } I0918 20:39:45.268358 28727 leader_election.cc:296] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [CANDIDATE]: Term 1 election: Requested vote from peers 02d4a9e1c4ac49fbbd722def96184901 (127.18.222.62:40573), 548b97f9463d4a3e9e3c60bf73bc1b97 (127.18.222.60:42325) I0918 20:39:45.269496 28694 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fc25e406a11476584aea076bc391255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" I0918 20:39:45.270284 28694 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:45.271404 28580 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fc25e406a11476584aea076bc391255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d4a9e1c4ac49fbbd722def96184901" I0918 20:39:45.272264 28580 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:45.275820 28694 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fc25e406a11476584aea076bc391255 in term 1. I0918 20:39:45.277071 28611 leader_election.cc:310] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548b97f9463d4a3e9e3c60bf73bc1b97, 6fc25e406a11476584aea076bc391255; no voters: I0918 20:39:45.278240 28727 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:45.279634 28727 raft_consensus.cc:667] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [term 1 LEADER]: Becoming Leader. State: Replica: 6fc25e406a11476584aea076bc391255, State: Running, Role: LEADER I0918 20:39:45.279616 28580 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fc25e406a11476584aea076bc391255 in term 1. I0918 20:39:45.280478 28727 consensus_queue.cc:206] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } I0918 20:39:45.282922 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:38499 I0918 20:39:45.286751 19320 webserver.cc:374] Webserver started at http://127.18.222.1:44203/ using document root and password file I0918 20:39:45.287444 28788 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fc25e406a11476584aea076bc391255. Latest consensus state: current_term: 1 leader_uuid: "6fc25e406a11476584aea076bc391255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } } I0918 20:39:45.288498 28788 sys_catalog.cc:343] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [sys.catalog]: This master's current role is: LEADER I0918 20:39:45.292827 28800 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:39:45.298015 28800 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:39:45.320232 28795 heartbeater.cc:322] Connected to a master server at 127.18.222.60:42325 I0918 20:39:45.320622 28795 heartbeater.cc:406] Registering TS with master... I0918 20:39:45.321310 28795 heartbeater.cc:455] Master 127.18.222.60:42325 requested a full tablet report, sending... I0918 20:39:45.322602 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 170092us I0918 20:39:45.322947 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:45.323083 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:45.323263 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:45.323367 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:45.323998 28796 heartbeater.cc:322] Connected to a master server at 127.18.222.61:40165 I0918 20:39:45.324434 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-1-root/instance: uuid: "925ca64f37c3497f91d3e02bd2d92385" format_stamp: "Formatted at 2019-09-18 20:39:45 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:45.324415 28796 heartbeater.cc:406] Registering TS with master... I0918 20:39:45.324900 28684 ts_manager.cc:186] Registered new tserver with Master: 7738715631eb46b18c9c15fa8afa48cd (127.18.222.1:38499) I0918 20:39:45.325610 28796 heartbeater.cc:455] Master 127.18.222.61:40165 requested a full tablet report, sending... I0918 20:39:45.327730 28627 ts_manager.cc:186] Registered new tserver with Master: 7738715631eb46b18c9c15fa8afa48cd (127.18.222.1:38499) I0918 20:39:45.331316 28798 heartbeater.cc:322] Connected to a master server at 127.18.222.62:40573 I0918 20:39:45.331636 28798 heartbeater.cc:406] Registering TS with master... I0918 20:39:45.332325 28798 heartbeater.cc:455] Master 127.18.222.62:40573 requested a full tablet report, sending... I0918 20:39:45.334815 28570 ts_manager.cc:186] Registered new tserver with Master: 7738715631eb46b18c9c15fa8afa48cd (127.18.222.1:38499) I0918 20:39:45.337416 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.012s user 0.010s sys 0.003s I0918 20:39:45.339143 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:45.350397 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s I0918 20:39:45.350606 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-1-root uuid: "925ca64f37c3497f91d3e02bd2d92385" format_stamp: "Formatted at 2019-09-18 20:39:45 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:45.350847 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:45.372543 28694 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97 [term 1 FOLLOWER]: Refusing update from remote peer 6fc25e406a11476584aea076bc391255: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:45.372625 28580 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901 [term 1 FOLLOWER]: Refusing update from remote peer 6fc25e406a11476584aea076bc391255: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:45.374107 28727 consensus_queue.cc:984] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:45.374850 28788 consensus_queue.cc:984] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:45.389403 28721 sys_catalog.cc:340] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fc25e406a11476584aea076bc391255. Latest consensus state: current_term: 1 leader_uuid: "6fc25e406a11476584aea076bc391255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } } I0918 20:39:45.390347 28721 sys_catalog.cc:343] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:45.392953 28724 sys_catalog.cc:340] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fc25e406a11476584aea076bc391255. Latest consensus state: current_term: 1 leader_uuid: "6fc25e406a11476584aea076bc391255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } } I0918 20:39:45.393983 28724 sys_catalog.cc:343] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:45.402740 28788 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fc25e406a11476584aea076bc391255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } } I0918 20:39:45.404783 28788 sys_catalog.cc:343] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [sys.catalog]: This master's current role is: LEADER I0918 20:39:45.404901 28724 sys_catalog.cc:340] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fc25e406a11476584aea076bc391255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } } I0918 20:39:45.405786 28724 sys_catalog.cc:343] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:45.406204 28727 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fc25e406a11476584aea076bc391255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } } I0918 20:39:45.407826 28727 sys_catalog.cc:343] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [sys.catalog]: This master's current role is: LEADER I0918 20:39:45.408198 28721 sys_catalog.cc:340] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fc25e406a11476584aea076bc391255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d4a9e1c4ac49fbbd722def96184901" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40573 } } peers { permanent_uuid: "6fc25e406a11476584aea076bc391255" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 40165 } } peers { permanent_uuid: "548b97f9463d4a3e9e3c60bf73bc1b97" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 42325 } } } I0918 20:39:45.409016 28721 sys_catalog.cc:343] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:45.410084 28800 catalog_manager.cc:981] Generated new certificate authority record I0918 20:39:45.412716 28800 catalog_manager.cc:1106] Loading token signing keys... I0918 20:39:45.432987 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:45.433290 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:45.441901 28800 catalog_manager.cc:4446] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255: Generated new TSK 0 I0918 20:39:45.442885 28800 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:39:45.460747 28627 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53908 I0918 20:39:45.475455 28796 heartbeater.cc:447] Master 127.18.222.61:40165 was elected leader, sending a full tablet report... I0918 20:39:45.509069 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.2:42837 I0918 20:39:45.511418 19320 webserver.cc:374] Webserver started at http://127.18.222.2:34271/ using document root and password file I0918 20:39:45.545936 28877 heartbeater.cc:322] Connected to a master server at 127.18.222.61:40165 I0918 20:39:45.546165 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 170092us I0918 20:39:45.546339 28877 heartbeater.cc:406] Registering TS with master... I0918 20:39:45.546694 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:45.546977 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:45.547000 28877 heartbeater.cc:455] Master 127.18.222.61:40165 requested a full tablet report, sending... I0918 20:39:45.547251 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:45.546986 28878 heartbeater.cc:322] Connected to a master server at 127.18.222.62:40573 I0918 20:39:45.547718 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:45.547919 28876 heartbeater.cc:322] Connected to a master server at 127.18.222.60:42325 I0918 20:39:45.548349 28876 heartbeater.cc:406] Registering TS with master... I0918 20:39:45.548892 28876 heartbeater.cc:455] Master 127.18.222.60:42325 requested a full tablet report, sending... I0918 20:39:45.549144 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-2-root/instance: uuid: "0700a64f81a645d6aa935fa436d4d9fe" format_stamp: "Formatted at 2019-09-18 20:39:45 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:45.549728 28627 ts_manager.cc:186] Registered new tserver with Master: 925ca64f37c3497f91d3e02bd2d92385 (127.18.222.2:42837) I0918 20:39:45.551334 28684 ts_manager.cc:186] Registered new tserver with Master: 925ca64f37c3497f91d3e02bd2d92385 (127.18.222.2:42837) I0918 20:39:45.547946 28878 heartbeater.cc:406] Registering TS with master... I0918 20:39:45.552394 28878 heartbeater.cc:455] Master 127.18.222.62:40573 requested a full tablet report, sending... I0918 20:39:45.553454 28627 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53914 I0918 20:39:45.555516 28570 ts_manager.cc:186] Registered new tserver with Master: 925ca64f37c3497f91d3e02bd2d92385 (127.18.222.2:42837) I0918 20:39:45.563897 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.014s user 0.015s sys 0.000s I0918 20:39:45.566462 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:45.567245 28877 heartbeater.cc:447] Master 127.18.222.61:40165 was elected leader, sending a full tablet report... I0918 20:39:45.578943 19320 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s I0918 20:39:45.579365 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-2-root uuid: "0700a64f81a645d6aa935fa436d4d9fe" format_stamp: "Formatted at 2019-09-18 20:39:45 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:45.579845 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:45.618213 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:45.618435 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:45.676481 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.3:38729 I0918 20:39:45.678963 19320 webserver.cc:374] Webserver started at http://127.18.222.3:37303/ using document root and password file I0918 20:39:45.707839 28948 heartbeater.cc:322] Connected to a master server at 127.18.222.61:40165 I0918 20:39:45.708338 28948 heartbeater.cc:406] Registering TS with master... I0918 20:39:45.709055 28948 heartbeater.cc:455] Master 127.18.222.61:40165 requested a full tablet report, sending... I0918 20:39:45.711539 28951 heartbeater.cc:322] Connected to a master server at 127.18.222.62:40573 I0918 20:39:45.712342 28951 heartbeater.cc:406] Registering TS with master... I0918 20:39:45.712906 28951 heartbeater.cc:455] Master 127.18.222.62:40573 requested a full tablet report, sending... I0918 20:39:45.715330 28626 ts_manager.cc:186] Registered new tserver with Master: 0700a64f81a645d6aa935fa436d4d9fe (127.18.222.3:38729) I0918 20:39:45.715682 28569 ts_manager.cc:186] Registered new tserver with Master: 0700a64f81a645d6aa935fa436d4d9fe (127.18.222.3:38729) I0918 20:39:45.716293 28947 heartbeater.cc:322] Connected to a master server at 127.18.222.60:42325 I0918 20:39:45.716697 28947 heartbeater.cc:406] Registering TS with master... I0918 20:39:45.717280 28947 heartbeater.cc:455] Master 127.18.222.60:42325 requested a full tablet report, sending... I0918 20:39:45.717454 28626 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53920 I0918 20:39:45.719616 28683 ts_manager.cc:186] Registered new tserver with Master: 0700a64f81a645d6aa935fa436d4d9fe (127.18.222.3:38729) I0918 20:39:45.721303 19320 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.033737596s I0918 20:39:45.731488 28948 heartbeater.cc:447] Master 127.18.222.61:40165 was elected leader, sending a full tablet report... I0918 20:39:45.784270 28627 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53926: name: "ksck-test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\n\000\000\000\004\001\024\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:39:45.786648 28627 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:39:45.908464 28843 tablet_service.cc:1022] Processing CreateTablet for tablet 522193bd3bc84e4ab6bb85c55516f0a9 (table=ksck-test-table [id=852db7d3af1542d2b0dc97327c0904ec]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:45.909258 28843 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 522193bd3bc84e4ab6bb85c55516f0a9. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:45.909271 28842 tablet_service.cc:1022] Processing CreateTablet for tablet 959e7115d2f6440fa1d0723f91a13d09 (table=ksck-test-table [id=852db7d3af1542d2b0dc97327c0904ec]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:45.910660 28842 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 959e7115d2f6440fa1d0723f91a13d09. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:45.911833 28760 tablet_service.cc:1022] Processing CreateTablet for tablet 522193bd3bc84e4ab6bb85c55516f0a9 (table=ksck-test-table [id=852db7d3af1542d2b0dc97327c0904ec]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:45.912374 28759 tablet_service.cc:1022] Processing CreateTablet for tablet 959e7115d2f6440fa1d0723f91a13d09 (table=ksck-test-table [id=852db7d3af1542d2b0dc97327c0904ec]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:45.912938 28760 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 522193bd3bc84e4ab6bb85c55516f0a9. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:45.912771 28841 tablet_service.cc:1022] Processing CreateTablet for tablet c8c0bb9a100d485d857e056963057a6c (table=ksck-test-table [id=852db7d3af1542d2b0dc97327c0904ec]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:45.913379 28759 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 959e7115d2f6440fa1d0723f91a13d09. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:45.913763 28841 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet c8c0bb9a100d485d857e056963057a6c. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:45.924333 28758 tablet_service.cc:1022] Processing CreateTablet for tablet c8c0bb9a100d485d857e056963057a6c (table=ksck-test-table [id=852db7d3af1542d2b0dc97327c0904ec]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:45.925581 28758 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet c8c0bb9a100d485d857e056963057a6c. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:45.960253 28968 tablet_bootstrap.cc:439] T 522193bd3bc84e4ab6bb85c55516f0a9 P 925ca64f37c3497f91d3e02bd2d92385: Bootstrap starting. I0918 20:39:45.960686 28914 tablet_service.cc:1022] Processing CreateTablet for tablet 522193bd3bc84e4ab6bb85c55516f0a9 (table=ksck-test-table [id=852db7d3af1542d2b0dc97327c0904ec]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:45.962766 28914 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 522193bd3bc84e4ab6bb85c55516f0a9. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:45.963656 28968 tablet_bootstrap.cc:590] T 522193bd3bc84e4ab6bb85c55516f0a9 P 925ca64f37c3497f91d3e02bd2d92385: No blocks or log segments found. Creating new log. I0918 20:39:45.965518 28969 tablet_bootstrap.cc:439] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd: Bootstrap starting. I0918 20:39:45.965844 28913 tablet_service.cc:1022] Processing CreateTablet for tablet 959e7115d2f6440fa1d0723f91a13d09 (table=ksck-test-table [id=852db7d3af1542d2b0dc97327c0904ec]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:45.966876 28913 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 959e7115d2f6440fa1d0723f91a13d09. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:45.974591 28912 tablet_service.cc:1022] Processing CreateTablet for tablet c8c0bb9a100d485d857e056963057a6c (table=ksck-test-table [id=852db7d3af1542d2b0dc97327c0904ec]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:45.975984 28969 tablet_bootstrap.cc:590] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd: No blocks or log segments found. Creating new log. I0918 20:39:45.976028 28912 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet c8c0bb9a100d485d857e056963057a6c. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:45.980969 28969 tablet_bootstrap.cc:439] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd: No bootstrap required, opened a new log I0918 20:39:45.981109 28971 tablet_bootstrap.cc:439] T 522193bd3bc84e4ab6bb85c55516f0a9 P 0700a64f81a645d6aa935fa436d4d9fe: Bootstrap starting. I0918 20:39:45.981619 28969 ts_tablet_manager.cc:1085] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.006s I0918 20:39:45.984308 28969 raft_consensus.cc:340] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } I0918 20:39:45.985102 28969 raft_consensus.cc:366] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:45.985348 28969 raft_consensus.cc:705] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7738715631eb46b18c9c15fa8afa48cd, State: Initialized, Role: FOLLOWER I0918 20:39:45.986177 28969 consensus_queue.cc:229] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } I0918 20:39:45.996820 28969 ts_tablet_manager.cc:1113] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd: Time spent starting tablet: real 0.015s user 0.010s sys 0.002s I0918 20:39:46.014186 28969 tablet_bootstrap.cc:439] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd: Bootstrap starting. I0918 20:39:46.018386 28969 tablet_bootstrap.cc:590] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd: No blocks or log segments found. Creating new log. I0918 20:39:46.023663 28971 tablet_bootstrap.cc:590] T 522193bd3bc84e4ab6bb85c55516f0a9 P 0700a64f81a645d6aa935fa436d4d9fe: No blocks or log segments found. Creating new log. I0918 20:39:46.024832 28968 tablet_bootstrap.cc:439] T 522193bd3bc84e4ab6bb85c55516f0a9 P 925ca64f37c3497f91d3e02bd2d92385: No bootstrap required, opened a new log I0918 20:39:46.025326 28968 ts_tablet_manager.cc:1085] T 522193bd3bc84e4ab6bb85c55516f0a9 P 925ca64f37c3497f91d3e02bd2d92385: Time spent bootstrapping tablet: real 0.065s user 0.007s sys 0.012s I0918 20:39:46.028717 28968 raft_consensus.cc:340] T 522193bd3bc84e4ab6bb85c55516f0a9 P 925ca64f37c3497f91d3e02bd2d92385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } I0918 20:39:46.029496 28968 raft_consensus.cc:366] T 522193bd3bc84e4ab6bb85c55516f0a9 P 925ca64f37c3497f91d3e02bd2d92385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:46.029731 28968 raft_consensus.cc:705] T 522193bd3bc84e4ab6bb85c55516f0a9 P 925ca64f37c3497f91d3e02bd2d92385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 925ca64f37c3497f91d3e02bd2d92385, State: Initialized, Role: FOLLOWER I0918 20:39:46.030756 28968 consensus_queue.cc:229] T 522193bd3bc84e4ab6bb85c55516f0a9 P 925ca64f37c3497f91d3e02bd2d92385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } I0918 20:39:46.033442 28969 tablet_bootstrap.cc:439] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd: No bootstrap required, opened a new log I0918 20:39:46.034005 28969 ts_tablet_manager.cc:1085] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.000s I0918 20:39:46.038733 28969 raft_consensus.cc:340] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } I0918 20:39:46.039482 28969 raft_consensus.cc:366] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:46.039721 28969 raft_consensus.cc:705] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7738715631eb46b18c9c15fa8afa48cd, State: Initialized, Role: FOLLOWER I0918 20:39:46.040371 28971 tablet_bootstrap.cc:439] T 522193bd3bc84e4ab6bb85c55516f0a9 P 0700a64f81a645d6aa935fa436d4d9fe: No bootstrap required, opened a new log I0918 20:39:46.041083 28971 ts_tablet_manager.cc:1085] T 522193bd3bc84e4ab6bb85c55516f0a9 P 0700a64f81a645d6aa935fa436d4d9fe: Time spent bootstrapping tablet: real 0.060s user 0.008s sys 0.000s I0918 20:39:46.043987 28969 consensus_queue.cc:229] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } I0918 20:39:46.055002 28969 ts_tablet_manager.cc:1113] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s I0918 20:39:46.055966 28969 tablet_bootstrap.cc:439] T 959e7115d2f6440fa1d0723f91a13d09 P 7738715631eb46b18c9c15fa8afa48cd: Bootstrap starting. I0918 20:39:46.048426 28968 ts_tablet_manager.cc:1113] T 522193bd3bc84e4ab6bb85c55516f0a9 P 925ca64f37c3497f91d3e02bd2d92385: Time spent starting tablet: real 0.023s user 0.003s sys 0.005s I0918 20:39:46.057493 28968 tablet_bootstrap.cc:439] T 959e7115d2f6440fa1d0723f91a13d09 P 925ca64f37c3497f91d3e02bd2d92385: Bootstrap starting. I0918 20:39:46.061328 28968 tablet_bootstrap.cc:590] T 959e7115d2f6440fa1d0723f91a13d09 P 925ca64f37c3497f91d3e02bd2d92385: No blocks or log segments found. Creating new log. I0918 20:39:46.069977 28968 tablet_bootstrap.cc:439] T 959e7115d2f6440fa1d0723f91a13d09 P 925ca64f37c3497f91d3e02bd2d92385: No bootstrap required, opened a new log I0918 20:39:46.070502 28968 ts_tablet_manager.cc:1085] T 959e7115d2f6440fa1d0723f91a13d09 P 925ca64f37c3497f91d3e02bd2d92385: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s I0918 20:39:46.072283 28968 raft_consensus.cc:340] T 959e7115d2f6440fa1d0723f91a13d09 P 925ca64f37c3497f91d3e02bd2d92385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } I0918 20:39:46.072804 28968 raft_consensus.cc:366] T 959e7115d2f6440fa1d0723f91a13d09 P 925ca64f37c3497f91d3e02bd2d92385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:46.072996 28968 raft_consensus.cc:705] T 959e7115d2f6440fa1d0723f91a13d09 P 925ca64f37c3497f91d3e02bd2d92385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 925ca64f37c3497f91d3e02bd2d92385, State: Initialized, Role: FOLLOWER I0918 20:39:46.043339 28971 raft_consensus.cc:340] T 522193bd3bc84e4ab6bb85c55516f0a9 P 0700a64f81a645d6aa935fa436d4d9fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } I0918 20:39:46.073547 28971 raft_consensus.cc:366] T 522193bd3bc84e4ab6bb85c55516f0a9 P 0700a64f81a645d6aa935fa436d4d9fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:46.073653 28968 consensus_queue.cc:229] T 959e7115d2f6440fa1d0723f91a13d09 P 925ca64f37c3497f91d3e02bd2d92385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } I0918 20:39:46.076608 28968 ts_tablet_manager.cc:1113] T 959e7115d2f6440fa1d0723f91a13d09 P 925ca64f37c3497f91d3e02bd2d92385: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s I0918 20:39:46.077582 28968 tablet_bootstrap.cc:439] T c8c0bb9a100d485d857e056963057a6c P 925ca64f37c3497f91d3e02bd2d92385: Bootstrap starting. I0918 20:39:46.082422 28969 tablet_bootstrap.cc:590] T 959e7115d2f6440fa1d0723f91a13d09 P 7738715631eb46b18c9c15fa8afa48cd: No blocks or log segments found. Creating new log. I0918 20:39:46.073741 28971 raft_consensus.cc:705] T 522193bd3bc84e4ab6bb85c55516f0a9 P 0700a64f81a645d6aa935fa436d4d9fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0700a64f81a645d6aa935fa436d4d9fe, State: Initialized, Role: FOLLOWER I0918 20:39:46.084873 28971 consensus_queue.cc:229] T 522193bd3bc84e4ab6bb85c55516f0a9 P 0700a64f81a645d6aa935fa436d4d9fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } I0918 20:39:46.098233 28969 tablet_bootstrap.cc:439] T 959e7115d2f6440fa1d0723f91a13d09 P 7738715631eb46b18c9c15fa8afa48cd: No bootstrap required, opened a new log I0918 20:39:46.098798 28969 ts_tablet_manager.cc:1085] T 959e7115d2f6440fa1d0723f91a13d09 P 7738715631eb46b18c9c15fa8afa48cd: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.001s I0918 20:39:46.101256 28969 raft_consensus.cc:340] T 959e7115d2f6440fa1d0723f91a13d09 P 7738715631eb46b18c9c15fa8afa48cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } I0918 20:39:46.101753 28969 raft_consensus.cc:366] T 959e7115d2f6440fa1d0723f91a13d09 P 7738715631eb46b18c9c15fa8afa48cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:46.101939 28969 raft_consensus.cc:705] T 959e7115d2f6440fa1d0723f91a13d09 P 7738715631eb46b18c9c15fa8afa48cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7738715631eb46b18c9c15fa8afa48cd, State: Initialized, Role: FOLLOWER I0918 20:39:46.102651 28968 tablet_bootstrap.cc:590] T c8c0bb9a100d485d857e056963057a6c P 925ca64f37c3497f91d3e02bd2d92385: No blocks or log segments found. Creating new log. I0918 20:39:46.102461 28969 consensus_queue.cc:229] T 959e7115d2f6440fa1d0723f91a13d09 P 7738715631eb46b18c9c15fa8afa48cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } I0918 20:39:46.110263 28722 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901: acquiring CA information for follower catalog manager: success I0918 20:39:46.114575 28722 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:46.121560 28969 ts_tablet_manager.cc:1113] T 959e7115d2f6440fa1d0723f91a13d09 P 7738715631eb46b18c9c15fa8afa48cd: Time spent starting tablet: real 0.022s user 0.004s sys 0.000s I0918 20:39:46.122395 28971 ts_tablet_manager.cc:1113] T 522193bd3bc84e4ab6bb85c55516f0a9 P 0700a64f81a645d6aa935fa436d4d9fe: Time spent starting tablet: real 0.081s user 0.011s sys 0.005s I0918 20:39:46.123342 28971 tablet_bootstrap.cc:439] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe: Bootstrap starting. I0918 20:39:46.125026 28968 tablet_bootstrap.cc:439] T c8c0bb9a100d485d857e056963057a6c P 925ca64f37c3497f91d3e02bd2d92385: No bootstrap required, opened a new log I0918 20:39:46.125639 28968 ts_tablet_manager.cc:1085] T c8c0bb9a100d485d857e056963057a6c P 925ca64f37c3497f91d3e02bd2d92385: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.008s I0918 20:39:46.128062 28968 raft_consensus.cc:340] T c8c0bb9a100d485d857e056963057a6c P 925ca64f37c3497f91d3e02bd2d92385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } I0918 20:39:46.128787 28968 raft_consensus.cc:366] T c8c0bb9a100d485d857e056963057a6c P 925ca64f37c3497f91d3e02bd2d92385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:46.129009 28968 raft_consensus.cc:705] T c8c0bb9a100d485d857e056963057a6c P 925ca64f37c3497f91d3e02bd2d92385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 925ca64f37c3497f91d3e02bd2d92385, State: Initialized, Role: FOLLOWER I0918 20:39:46.129792 28968 consensus_queue.cc:229] T c8c0bb9a100d485d857e056963057a6c P 925ca64f37c3497f91d3e02bd2d92385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } I0918 20:39:46.128584 28971 tablet_bootstrap.cc:590] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe: No blocks or log segments found. Creating new log. I0918 20:39:46.131765 28968 ts_tablet_manager.cc:1113] T c8c0bb9a100d485d857e056963057a6c P 925ca64f37c3497f91d3e02bd2d92385: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s I0918 20:39:46.135596 28971 tablet_bootstrap.cc:439] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe: No bootstrap required, opened a new log I0918 20:39:46.136296 28971 ts_tablet_manager.cc:1085] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s I0918 20:39:46.138721 28971 raft_consensus.cc:340] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } I0918 20:39:46.139516 28971 raft_consensus.cc:366] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:46.139792 28971 raft_consensus.cc:705] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0700a64f81a645d6aa935fa436d4d9fe, State: Initialized, Role: FOLLOWER I0918 20:39:46.140471 28971 consensus_queue.cc:229] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } I0918 20:39:46.142199 28971 ts_tablet_manager.cc:1113] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s I0918 20:39:46.142453 28725 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97: acquiring CA information for follower catalog manager: success I0918 20:39:46.143350 28971 tablet_bootstrap.cc:439] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe: Bootstrap starting. I0918 20:39:46.147627 28971 tablet_bootstrap.cc:590] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe: No blocks or log segments found. Creating new log. I0918 20:39:46.151630 28971 tablet_bootstrap.cc:439] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe: No bootstrap required, opened a new log I0918 20:39:46.152210 28971 ts_tablet_manager.cc:1085] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s I0918 20:39:46.155361 28971 raft_consensus.cc:340] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } I0918 20:39:46.156157 28971 raft_consensus.cc:366] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:46.156373 28971 raft_consensus.cc:705] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0700a64f81a645d6aa935fa436d4d9fe, State: Initialized, Role: FOLLOWER I0918 20:39:46.156993 28971 consensus_queue.cc:229] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } I0918 20:39:46.159037 28971 ts_tablet_manager.cc:1113] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s I0918 20:39:46.163424 28725 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:46.230358 28978 raft_consensus.cc:472] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:46.231050 28978 raft_consensus.cc:494] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } I0918 20:39:46.232882 28978 leader_election.cc:296] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7738715631eb46b18c9c15fa8afa48cd (127.18.222.1:38499), 925ca64f37c3497f91d3e02bd2d92385 (127.18.222.2:42837) I0918 20:39:46.262583 28853 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "959e7115d2f6440fa1d0723f91a13d09" candidate_uuid: "0700a64f81a645d6aa935fa436d4d9fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "925ca64f37c3497f91d3e02bd2d92385" is_pre_election: true I0918 20:39:46.263384 28853 raft_consensus.cc:2350] T 959e7115d2f6440fa1d0723f91a13d09 P 925ca64f37c3497f91d3e02bd2d92385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0700a64f81a645d6aa935fa436d4d9fe in term 0. I0918 20:39:46.264843 28886 leader_election.cc:310] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0700a64f81a645d6aa935fa436d4d9fe, 925ca64f37c3497f91d3e02bd2d92385; no voters: I0918 20:39:46.266752 28978 raft_consensus.cc:2644] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:46.267132 28978 raft_consensus.cc:472] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:46.267457 28978 raft_consensus.cc:2890] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:46.277397 28978 raft_consensus.cc:494] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } I0918 20:39:46.280645 28770 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "959e7115d2f6440fa1d0723f91a13d09" candidate_uuid: "0700a64f81a645d6aa935fa436d4d9fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7738715631eb46b18c9c15fa8afa48cd" is_pre_election: true I0918 20:39:46.281613 28770 raft_consensus.cc:2350] T 959e7115d2f6440fa1d0723f91a13d09 P 7738715631eb46b18c9c15fa8afa48cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0700a64f81a645d6aa935fa436d4d9fe in term 0. I0918 20:39:46.288300 28978 leader_election.cc:296] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe [CANDIDATE]: Term 1 election: Requested vote from peers 7738715631eb46b18c9c15fa8afa48cd (127.18.222.1:38499), 925ca64f37c3497f91d3e02bd2d92385 (127.18.222.2:42837) I0918 20:39:46.288797 28770 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "959e7115d2f6440fa1d0723f91a13d09" candidate_uuid: "0700a64f81a645d6aa935fa436d4d9fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7738715631eb46b18c9c15fa8afa48cd" I0918 20:39:46.289796 28853 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "959e7115d2f6440fa1d0723f91a13d09" candidate_uuid: "0700a64f81a645d6aa935fa436d4d9fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "925ca64f37c3497f91d3e02bd2d92385" I0918 20:39:46.290387 28770 raft_consensus.cc:2890] T 959e7115d2f6440fa1d0723f91a13d09 P 7738715631eb46b18c9c15fa8afa48cd [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:46.290462 28853 raft_consensus.cc:2890] T 959e7115d2f6440fa1d0723f91a13d09 P 925ca64f37c3497f91d3e02bd2d92385 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:46.298051 28853 raft_consensus.cc:2350] T 959e7115d2f6440fa1d0723f91a13d09 P 925ca64f37c3497f91d3e02bd2d92385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0700a64f81a645d6aa935fa436d4d9fe in term 1. I0918 20:39:46.298146 28770 raft_consensus.cc:2350] T 959e7115d2f6440fa1d0723f91a13d09 P 7738715631eb46b18c9c15fa8afa48cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0700a64f81a645d6aa935fa436d4d9fe in term 1. I0918 20:39:46.299350 28886 leader_election.cc:310] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0700a64f81a645d6aa935fa436d4d9fe, 925ca64f37c3497f91d3e02bd2d92385; no voters: I0918 20:39:46.300436 28978 raft_consensus.cc:2644] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:46.306344 28978 raft_consensus.cc:667] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe [term 1 LEADER]: Becoming Leader. State: Replica: 0700a64f81a645d6aa935fa436d4d9fe, State: Running, Role: LEADER I0918 20:39:46.307139 28978 consensus_queue.cc:206] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } I0918 20:39:46.320020 28626 catalog_manager.cc:4131] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe reported cstate change: term changed from 0 to 1, leader changed from to 0700a64f81a645d6aa935fa436d4d9fe (127.18.222.3). New cstate: current_term: 1 leader_uuid: "0700a64f81a645d6aa935fa436d4d9fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } health_report { overall_health: HEALTHY } } } I0918 20:39:46.374078 28972 raft_consensus.cc:472] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:46.374616 28972 raft_consensus.cc:494] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } I0918 20:39:46.376694 28972 leader_election.cc:296] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0700a64f81a645d6aa935fa436d4d9fe (127.18.222.3:38729), 925ca64f37c3497f91d3e02bd2d92385 (127.18.222.2:42837) I0918 20:39:46.392567 28976 raft_consensus.cc:472] T 522193bd3bc84e4ab6bb85c55516f0a9 P 925ca64f37c3497f91d3e02bd2d92385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:46.393362 28976 raft_consensus.cc:494] T 522193bd3bc84e4ab6bb85c55516f0a9 P 925ca64f37c3497f91d3e02bd2d92385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } I0918 20:39:46.395678 28976 leader_election.cc:296] T 522193bd3bc84e4ab6bb85c55516f0a9 P 925ca64f37c3497f91d3e02bd2d92385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0700a64f81a645d6aa935fa436d4d9fe (127.18.222.3:38729), 7738715631eb46b18c9c15fa8afa48cd (127.18.222.1:38499) I0918 20:39:46.408574 28972 raft_consensus.cc:472] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:46.412147 28972 raft_consensus.cc:494] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } I0918 20:39:46.414665 28972 leader_election.cc:296] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 925ca64f37c3497f91d3e02bd2d92385 (127.18.222.2:42837), 0700a64f81a645d6aa935fa436d4d9fe (127.18.222.3:38729) I0918 20:39:46.422632 28978 raft_consensus.cc:472] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:46.423463 28978 raft_consensus.cc:494] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } I0918 20:39:46.426851 28853 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c8c0bb9a100d485d857e056963057a6c" candidate_uuid: "0700a64f81a645d6aa935fa436d4d9fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "925ca64f37c3497f91d3e02bd2d92385" is_pre_election: true I0918 20:39:46.427608 28853 raft_consensus.cc:2350] T c8c0bb9a100d485d857e056963057a6c P 925ca64f37c3497f91d3e02bd2d92385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0700a64f81a645d6aa935fa436d4d9fe in term 0. I0918 20:39:46.428244 28978 leader_election.cc:296] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7738715631eb46b18c9c15fa8afa48cd (127.18.222.1:38499), 925ca64f37c3497f91d3e02bd2d92385 (127.18.222.2:42837) I0918 20:39:46.430393 28852 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "522193bd3bc84e4ab6bb85c55516f0a9" candidate_uuid: "7738715631eb46b18c9c15fa8afa48cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "925ca64f37c3497f91d3e02bd2d92385" is_pre_election: true I0918 20:39:46.431224 28852 raft_consensus.cc:2350] T 522193bd3bc84e4ab6bb85c55516f0a9 P 925ca64f37c3497f91d3e02bd2d92385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7738715631eb46b18c9c15fa8afa48cd in term 0. I0918 20:39:46.432974 28852 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c8c0bb9a100d485d857e056963057a6c" candidate_uuid: "7738715631eb46b18c9c15fa8afa48cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "925ca64f37c3497f91d3e02bd2d92385" is_pre_election: true I0918 20:39:46.433744 28852 raft_consensus.cc:2350] T c8c0bb9a100d485d857e056963057a6c P 925ca64f37c3497f91d3e02bd2d92385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7738715631eb46b18c9c15fa8afa48cd in term 0. I0918 20:39:46.434260 28886 leader_election.cc:310] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0700a64f81a645d6aa935fa436d4d9fe, 925ca64f37c3497f91d3e02bd2d92385; no voters: I0918 20:39:46.435199 28732 leader_election.cc:310] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7738715631eb46b18c9c15fa8afa48cd, 925ca64f37c3497f91d3e02bd2d92385; no voters: I0918 20:39:46.435423 28978 raft_consensus.cc:2644] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:46.435922 28978 raft_consensus.cc:472] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:46.436153 28972 raft_consensus.cc:2644] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:46.436281 28978 raft_consensus.cc:2890] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:46.436679 28972 raft_consensus.cc:472] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:46.437108 28972 raft_consensus.cc:2890] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:46.443406 28972 raft_consensus.cc:494] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } I0918 20:39:46.444823 28732 leader_election.cc:310] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7738715631eb46b18c9c15fa8afa48cd, 925ca64f37c3497f91d3e02bd2d92385; no voters: I0918 20:39:46.443226 28978 raft_consensus.cc:494] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } I0918 20:39:46.460201 28924 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "522193bd3bc84e4ab6bb85c55516f0a9" candidate_uuid: "7738715631eb46b18c9c15fa8afa48cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0700a64f81a645d6aa935fa436d4d9fe" is_pre_election: true I0918 20:39:46.462659 28923 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c8c0bb9a100d485d857e056963057a6c" candidate_uuid: "7738715631eb46b18c9c15fa8afa48cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0700a64f81a645d6aa935fa436d4d9fe" is_pre_election: true I0918 20:39:46.463582 28924 raft_consensus.cc:2350] T 522193bd3bc84e4ab6bb85c55516f0a9 P 0700a64f81a645d6aa935fa436d4d9fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7738715631eb46b18c9c15fa8afa48cd in term 0. I0918 20:39:46.469422 28923 raft_consensus.cc:2280] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7738715631eb46b18c9c15fa8afa48cd in current term 1: Already voted for candidate 0700a64f81a645d6aa935fa436d4d9fe in this term. I0918 20:39:46.472182 28972 leader_election.cc:296] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [CANDIDATE]: Term 1 election: Requested vote from peers 0700a64f81a645d6aa935fa436d4d9fe (127.18.222.3:38729), 925ca64f37c3497f91d3e02bd2d92385 (127.18.222.2:42837) I0918 20:39:46.472821 28972 raft_consensus.cc:2644] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:46.473105 28972 raft_consensus.cc:472] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:46.473428 28853 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c8c0bb9a100d485d857e056963057a6c" candidate_uuid: "0700a64f81a645d6aa935fa436d4d9fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "925ca64f37c3497f91d3e02bd2d92385" I0918 20:39:46.474143 28853 raft_consensus.cc:2890] T c8c0bb9a100d485d857e056963057a6c P 925ca64f37c3497f91d3e02bd2d92385 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:46.474095 28770 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c8c0bb9a100d485d857e056963057a6c" candidate_uuid: "0700a64f81a645d6aa935fa436d4d9fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7738715631eb46b18c9c15fa8afa48cd" is_pre_election: true I0918 20:39:46.473400 28972 raft_consensus.cc:2890] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:46.474035 28923 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "522193bd3bc84e4ab6bb85c55516f0a9" candidate_uuid: "7738715631eb46b18c9c15fa8afa48cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0700a64f81a645d6aa935fa436d4d9fe" I0918 20:39:46.476399 28923 raft_consensus.cc:2890] T 522193bd3bc84e4ab6bb85c55516f0a9 P 0700a64f81a645d6aa935fa436d4d9fe [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:46.483021 28852 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "522193bd3bc84e4ab6bb85c55516f0a9" candidate_uuid: "7738715631eb46b18c9c15fa8afa48cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "925ca64f37c3497f91d3e02bd2d92385" I0918 20:39:46.483837 28852 raft_consensus.cc:2890] T 522193bd3bc84e4ab6bb85c55516f0a9 P 925ca64f37c3497f91d3e02bd2d92385 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:46.490326 28852 raft_consensus.cc:2350] T 522193bd3bc84e4ab6bb85c55516f0a9 P 925ca64f37c3497f91d3e02bd2d92385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7738715631eb46b18c9c15fa8afa48cd in term 1. I0918 20:39:46.493533 28853 raft_consensus.cc:2350] T c8c0bb9a100d485d857e056963057a6c P 925ca64f37c3497f91d3e02bd2d92385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0700a64f81a645d6aa935fa436d4d9fe in term 1. I0918 20:39:46.494904 28886 leader_election.cc:310] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0700a64f81a645d6aa935fa436d4d9fe, 925ca64f37c3497f91d3e02bd2d92385; no voters: I0918 20:39:46.476109 28978 leader_election.cc:296] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [CANDIDATE]: Term 1 election: Requested vote from peers 7738715631eb46b18c9c15fa8afa48cd (127.18.222.1:38499), 925ca64f37c3497f91d3e02bd2d92385 (127.18.222.2:42837) I0918 20:39:46.493916 28923 raft_consensus.cc:2350] T 522193bd3bc84e4ab6bb85c55516f0a9 P 0700a64f81a645d6aa935fa436d4d9fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7738715631eb46b18c9c15fa8afa48cd in term 1. I0918 20:39:46.495936 28985 raft_consensus.cc:2644] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:46.500918 28924 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "522193bd3bc84e4ab6bb85c55516f0a9" candidate_uuid: "925ca64f37c3497f91d3e02bd2d92385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0700a64f81a645d6aa935fa436d4d9fe" is_pre_election: true I0918 20:39:46.502110 28924 raft_consensus.cc:2325] T 522193bd3bc84e4ab6bb85c55516f0a9 P 0700a64f81a645d6aa935fa436d4d9fe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 925ca64f37c3497f91d3e02bd2d92385 for term 1 because replica is already servicing an update from a current leader or another vote. I0918 20:39:46.503645 28733 leader_election.cc:310] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0700a64f81a645d6aa935fa436d4d9fe, 7738715631eb46b18c9c15fa8afa48cd; no voters: I0918 20:39:46.504779 28998 raft_consensus.cc:2644] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:46.507449 28998 raft_consensus.cc:667] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [term 1 LEADER]: Becoming Leader. State: Replica: 7738715631eb46b18c9c15fa8afa48cd, State: Running, Role: LEADER I0918 20:39:46.506902 28769 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "522193bd3bc84e4ab6bb85c55516f0a9" candidate_uuid: "925ca64f37c3497f91d3e02bd2d92385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7738715631eb46b18c9c15fa8afa48cd" is_pre_election: true I0918 20:39:46.509621 28768 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c8c0bb9a100d485d857e056963057a6c" candidate_uuid: "0700a64f81a645d6aa935fa436d4d9fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7738715631eb46b18c9c15fa8afa48cd" I0918 20:39:46.510704 28998 consensus_queue.cc:206] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } I0918 20:39:46.495062 28972 raft_consensus.cc:494] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } I0918 20:39:46.513164 28770 raft_consensus.cc:2280] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0700a64f81a645d6aa935fa436d4d9fe in current term 1: Already voted for candidate 7738715631eb46b18c9c15fa8afa48cd in this term. I0918 20:39:46.514483 28768 raft_consensus.cc:2325] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0700a64f81a645d6aa935fa436d4d9fe for term 1 because replica is already servicing an update from a current leader or another vote. I0918 20:39:46.512477 28985 raft_consensus.cc:667] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [term 1 LEADER]: Becoming Leader. State: Replica: 0700a64f81a645d6aa935fa436d4d9fe, State: Running, Role: LEADER I0918 20:39:46.517064 28985 consensus_queue.cc:206] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } } I0918 20:39:46.520498 28923 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c8c0bb9a100d485d857e056963057a6c" candidate_uuid: "7738715631eb46b18c9c15fa8afa48cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0700a64f81a645d6aa935fa436d4d9fe" I0918 20:39:46.520702 28852 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c8c0bb9a100d485d857e056963057a6c" candidate_uuid: "7738715631eb46b18c9c15fa8afa48cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "925ca64f37c3497f91d3e02bd2d92385" I0918 20:39:46.521489 28852 raft_consensus.cc:2280] T c8c0bb9a100d485d857e056963057a6c P 925ca64f37c3497f91d3e02bd2d92385 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7738715631eb46b18c9c15fa8afa48cd in current term 1: Already voted for candidate 0700a64f81a645d6aa935fa436d4d9fe in this term. I0918 20:39:46.519414 28972 leader_election.cc:296] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd [CANDIDATE]: Term 1 election: Requested vote from peers 925ca64f37c3497f91d3e02bd2d92385 (127.18.222.2:42837), 0700a64f81a645d6aa935fa436d4d9fe (127.18.222.3:38729) I0918 20:39:46.526648 28769 raft_consensus.cc:2311] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 925ca64f37c3497f91d3e02bd2d92385 for term 1 because replica is either leader or believes a valid leader to be alive. I0918 20:39:46.529376 28808 leader_election.cc:310] T 522193bd3bc84e4ab6bb85c55516f0a9 P 925ca64f37c3497f91d3e02bd2d92385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 925ca64f37c3497f91d3e02bd2d92385; no voters: 0700a64f81a645d6aa935fa436d4d9fe, 7738715631eb46b18c9c15fa8afa48cd I0918 20:39:46.533682 28976 raft_consensus.cc:2592] T 522193bd3bc84e4ab6bb85c55516f0a9 P 925ca64f37c3497f91d3e02bd2d92385 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0918 20:39:46.535421 28923 raft_consensus.cc:2311] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [term 1 LEADER]: Leader election vote request: Denying vote to candidate 7738715631eb46b18c9c15fa8afa48cd for term 1 because replica is either leader or believes a valid leader to be alive. I0918 20:39:46.536948 28733 leader_election.cc:310] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7738715631eb46b18c9c15fa8afa48cd; no voters: 0700a64f81a645d6aa935fa436d4d9fe, 925ca64f37c3497f91d3e02bd2d92385 I0918 20:39:46.532975 28626 catalog_manager.cc:4131] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd reported cstate change: term changed from 0 to 1, leader changed from to 7738715631eb46b18c9c15fa8afa48cd (127.18.222.1). New cstate: current_term: 1 leader_uuid: "7738715631eb46b18c9c15fa8afa48cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } health_report { overall_health: UNKNOWN } } } I0918 20:39:46.538353 28998 raft_consensus.cc:2592] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority I0918 20:39:46.538530 28627 catalog_manager.cc:4131] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe reported cstate change: term changed from 0 to 1, leader changed from to 0700a64f81a645d6aa935fa436d4d9fe (127.18.222.3). New cstate: current_term: 1 leader_uuid: "0700a64f81a645d6aa935fa436d4d9fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 } health_report { overall_health: HEALTHY } } } I0918 20:39:46.739645 28985 consensus_queue.cc:984] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:46.797435 28978 consensus_queue.cc:984] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:46.942142 28978 consensus_queue.cc:984] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "7738715631eb46b18c9c15fa8afa48cd" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:46.970286 28985 consensus_queue.cc:984] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:47.097003 28998 consensus_queue.cc:984] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "925ca64f37c3497f91d3e02bd2d92385" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 42837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:47.175248 28998 consensus_queue.cc:984] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0700a64f81a645d6aa935fa436d4d9fe" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 38729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:47.246166 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:47.332322 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:47.332727 19320 tablet_replica.cc:273] T 959e7115d2f6440fa1d0723f91a13d09 P 7738715631eb46b18c9c15fa8afa48cd: stopping tablet replica I0918 20:39:47.333117 19320 raft_consensus.cc:2147] T 959e7115d2f6440fa1d0723f91a13d09 P 7738715631eb46b18c9c15fa8afa48cd [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:47.333422 19320 raft_consensus.cc:2174] T 959e7115d2f6440fa1d0723f91a13d09 P 7738715631eb46b18c9c15fa8afa48cd [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:47.336175 19320 tablet_replica.cc:273] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd: stopping tablet replica I0918 20:39:47.336565 19320 raft_consensus.cc:2147] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:47.336844 19320 raft_consensus.cc:2174] T c8c0bb9a100d485d857e056963057a6c P 7738715631eb46b18c9c15fa8afa48cd [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:47.339517 19320 tablet_replica.cc:273] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd: stopping tablet replica I0918 20:39:47.340083 19320 raft_consensus.cc:2147] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:47.341012 19320 raft_consensus.cc:2174] T 522193bd3bc84e4ab6bb85c55516f0a9 P 7738715631eb46b18c9c15fa8afa48cd [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:47.365459 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:47.365741 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:47.366124 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:47.378811 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:47.391258 28805 messenger.cc:381] Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer W0918 20:39:47.392417 28886 consensus_peers.cc:458] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe -> Peer 925ca64f37c3497f91d3e02bd2d92385 (127.18.222.2:42837): Couldn't send request to peer 925ca64f37c3497f91d3e02bd2d92385. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry. W0918 20:39:47.408659 28888 consensus_peers.cc:458] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe -> Peer 7738715631eb46b18c9c15fa8afa48cd (127.18.222.1:38499): Couldn't send request to peer 7738715631eb46b18c9c15fa8afa48cd. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.1:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:39:47.427911 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:47.428392 19320 tablet_replica.cc:273] T 959e7115d2f6440fa1d0723f91a13d09 P 925ca64f37c3497f91d3e02bd2d92385: stopping tablet replica I0918 20:39:47.428856 19320 raft_consensus.cc:2147] T 959e7115d2f6440fa1d0723f91a13d09 P 925ca64f37c3497f91d3e02bd2d92385 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:47.429247 19320 raft_consensus.cc:2174] T 959e7115d2f6440fa1d0723f91a13d09 P 925ca64f37c3497f91d3e02bd2d92385 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:47.432709 19320 tablet_replica.cc:273] T c8c0bb9a100d485d857e056963057a6c P 925ca64f37c3497f91d3e02bd2d92385: stopping tablet replica I0918 20:39:47.433106 19320 raft_consensus.cc:2147] T c8c0bb9a100d485d857e056963057a6c P 925ca64f37c3497f91d3e02bd2d92385 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:47.433378 19320 raft_consensus.cc:2174] T c8c0bb9a100d485d857e056963057a6c P 925ca64f37c3497f91d3e02bd2d92385 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:47.436349 19320 tablet_replica.cc:273] T 522193bd3bc84e4ab6bb85c55516f0a9 P 925ca64f37c3497f91d3e02bd2d92385: stopping tablet replica I0918 20:39:47.436714 19320 raft_consensus.cc:2147] T 522193bd3bc84e4ab6bb85c55516f0a9 P 925ca64f37c3497f91d3e02bd2d92385 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:47.436981 19320 raft_consensus.cc:2174] T 522193bd3bc84e4ab6bb85c55516f0a9 P 925ca64f37c3497f91d3e02bd2d92385 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:47.453680 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:47.453881 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:47.454229 19320 tablet_server.cc:169] TabletServer shutdown complete. W0918 20:39:47.465909 28888 consensus_peers.cc:458] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe -> Peer 7738715631eb46b18c9c15fa8afa48cd (127.18.222.1:38499): Couldn't send request to peer 7738715631eb46b18c9c15fa8afa48cd. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.1:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:39:47.467125 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:47.501298 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:47.501682 19320 tablet_replica.cc:273] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe: stopping tablet replica I0918 20:39:47.502171 19320 raft_consensus.cc:2147] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:47.502979 19320 raft_consensus.cc:2174] T 959e7115d2f6440fa1d0723f91a13d09 P 0700a64f81a645d6aa935fa436d4d9fe [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:47.505527 19320 tablet_replica.cc:273] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe: stopping tablet replica I0918 20:39:47.505889 19320 raft_consensus.cc:2147] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:47.506516 19320 raft_consensus.cc:2174] T c8c0bb9a100d485d857e056963057a6c P 0700a64f81a645d6aa935fa436d4d9fe [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:47.509155 19320 tablet_replica.cc:273] T 522193bd3bc84e4ab6bb85c55516f0a9 P 0700a64f81a645d6aa935fa436d4d9fe: stopping tablet replica I0918 20:39:47.509521 19320 raft_consensus.cc:2147] T 522193bd3bc84e4ab6bb85c55516f0a9 P 0700a64f81a645d6aa935fa436d4d9fe [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:47.509790 19320 raft_consensus.cc:2174] T 522193bd3bc84e4ab6bb85c55516f0a9 P 0700a64f81a645d6aa935fa436d4d9fe [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:47.533558 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:47.533784 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:47.534133 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:47.544911 19320 master.cc:247] Master@127.18.222.62:40573 shutting down... I0918 20:39:47.562695 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:47.563166 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:47.563422 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 02d4a9e1c4ac49fbbd722def96184901: stopping tablet replica I0918 20:39:47.582583 19320 master.cc:258] Master@127.18.222.62:40573 shutdown complete. I0918 20:39:47.593600 19320 master.cc:247] Master@127.18.222.61:40165 shutting down... I0918 20:39:47.611052 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:47.611863 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:47.612121 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 6fc25e406a11476584aea076bc391255: stopping tablet replica I0918 20:39:47.630973 19320 master.cc:258] Master@127.18.222.61:40165 shutdown complete. I0918 20:39:47.642027 19320 master.cc:247] Master@127.18.222.60:42325 shutting down... I0918 20:39:47.659202 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:47.659581 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:47.659858 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 548b97f9463d4a3e9e3c60bf73bc1b97: stopping tablet replica I0918 20:39:47.677932 19320 master.cc:258] Master@127.18.222.60:42325 shutdown complete. [ OK ] RemoteKsckTest.TestTableFiltersNoMatch (2991 ms) [ RUN ] RemoteKsckTest.TestTableFilters I0918 20:39:47.708566 19320 test_util.cc:212] Using random seed: -696370565 I0918 20:39:47.709329 19320 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.18.222.62:40629,127.18.222.61:39277,127.18.222.60:39727 I0918 20:39:47.721422 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 171092us I0918 20:39:47.721801 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:47.721946 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:47.722200 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:47.722332 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:47.723348 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/master-0-root/instance: uuid: "38753c8ea6244f958281a39c906cd411" format_stamp: "Formatted at 2019-09-18 20:39:47 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:47.732034 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s I0918 20:39:47.733582 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:47.740267 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:47.740442 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/master-0-root uuid: "38753c8ea6244f958281a39c906cd411" format_stamp: "Formatted at 2019-09-18 20:39:47 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:47.740711 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:47.815178 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:40629 I0918 20:39:47.816962 19320 webserver.cc:374] Webserver started at http://127.18.222.62:42323/ using document root and password file I0918 20:39:47.818704 29103 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:47.823632 29103 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:47.833406 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 171092us I0918 20:39:47.833813 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:47.833971 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:47.834213 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:47.834360 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:47.835842 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/master-1-root/instance: uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" format_stamp: "Formatted at 2019-09-18 20:39:47 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:47.838846 29103 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:47.840764 29103 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:39277: Network error: Client connection negotiation failed: client connection to 127.18.222.61:39277: connect: Connection refused (error 111) I0918 20:39:47.846683 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.010s sys 0.000s I0918 20:39:47.848541 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:47.856353 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:47.856547 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/master-1-root uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" format_stamp: "Formatted at 2019-09-18 20:39:47 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:47.856825 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:47.868170 29103 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } attempt: 1 W0918 20:39:47.870365 29103 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:39277: Network error: Client connection negotiation failed: client connection to 127.18.222.61:39277: connect: Connection refused (error 111) I0918 20:39:47.913797 29103 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } attempt: 2 W0918 20:39:47.915941 29103 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:39277: Network error: Client connection negotiation failed: client connection to 127.18.222.61:39277: connect: Connection refused (error 111) I0918 20:39:47.965375 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:39277 I0918 20:39:47.967170 19320 webserver.cc:374] Webserver started at http://127.18.222.61:37421/ using document root and password file I0918 20:39:47.969091 29160 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:47.974431 29160 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:47.988536 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 171092us I0918 20:39:47.988994 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:47.989225 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:47.989503 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:47.989668 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:47.990844 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/master-2-root/instance: uuid: "50f4f536877e4dc3b4f97c7d7a214cac" format_stamp: "Formatted at 2019-09-18 20:39:47 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:47.991153 29160 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:48.003916 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.013s user 0.014s sys 0.001s I0918 20:39:48.005400 29103 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } attempt: 3 I0918 20:39:48.006458 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:48.014202 29160 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:48.016850 29160 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:39727: Network error: Client connection negotiation failed: client connection to 127.18.222.60:39727: connect: Connection refused (error 111) I0918 20:39:48.018007 29103 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:48.019834 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:39:48.020154 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/master-2-root uuid: "50f4f536877e4dc3b4f97c7d7a214cac" format_stamp: "Formatted at 2019-09-18 20:39:47 on dist-test-slave-dist-test-slave-85lk" W0918 20:39:48.020300 29103 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:39727: Network error: Client connection negotiation failed: client connection to 127.18.222.60:39727: connect: Connection refused (error 111) I0918 20:39:48.020454 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:48.051321 29160 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } attempt: 1 W0918 20:39:48.053588 29160 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:39727: Network error: Client connection negotiation failed: client connection to 127.18.222.60:39727: connect: Connection refused (error 111) I0918 20:39:48.063987 29103 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } attempt: 1 W0918 20:39:48.066653 29103 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:39727: Network error: Client connection negotiation failed: client connection to 127.18.222.60:39727: connect: Connection refused (error 111) I0918 20:39:48.078692 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:39727 I0918 20:39:48.080720 19320 webserver.cc:374] Webserver started at http://127.18.222.60:37573/ using document root and password file I0918 20:39:48.082504 29218 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:48.086992 29160 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } attempt: 2 I0918 20:39:48.087843 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0918 20:39:48.088275 29218 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:48.103139 29218 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:48.107358 29160 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a: Bootstrap starting. I0918 20:39:48.110941 29160 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a: No blocks or log segments found. Creating new log. I0918 20:39:48.114598 29160 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a: No bootstrap required, opened a new log I0918 20:39:48.114987 29218 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:48.116664 29160 raft_consensus.cc:340] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } I0918 20:39:48.117192 29160 raft_consensus.cc:366] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:48.117383 29160 raft_consensus.cc:705] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e0a29e7c61e40d0a4293b62ec0ac59a, State: Initialized, Role: FOLLOWER I0918 20:39:48.118067 29160 consensus_queue.cc:229] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } I0918 20:39:48.119786 29160 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } } I0918 20:39:48.120507 29160 sys_catalog.cc:343] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:48.119982 29222 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } } I0918 20:39:48.120853 29222 sys_catalog.cc:343] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:48.121628 29160 sys_catalog.cc:428] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:48.122205 29103 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } attempt: 2 W0918 20:39:48.128253 29223 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:48.128574 29223 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:48.135432 29218 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac: Bootstrap starting. I0918 20:39:48.138263 29218 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac: No blocks or log segments found. Creating new log. I0918 20:39:48.138545 29103 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411: Bootstrap starting. I0918 20:39:48.141248 29103 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411: No blocks or log segments found. Creating new log. I0918 20:39:48.141366 29218 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac: No bootstrap required, opened a new log I0918 20:39:48.143242 29218 raft_consensus.cc:340] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } I0918 20:39:48.144497 29218 raft_consensus.cc:366] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:48.144666 29218 raft_consensus.cc:705] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50f4f536877e4dc3b4f97c7d7a214cac, State: Initialized, Role: FOLLOWER I0918 20:39:48.145102 29103 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411: No bootstrap required, opened a new log I0918 20:39:48.145197 29218 consensus_queue.cc:229] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } I0918 20:39:48.147064 29103 raft_consensus.cc:340] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } I0918 20:39:48.147577 29103 raft_consensus.cc:366] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:48.148100 29103 raft_consensus.cc:705] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38753c8ea6244f958281a39c906cd411, State: Initialized, Role: FOLLOWER I0918 20:39:48.148249 29218 sys_catalog.cc:340] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } } I0918 20:39:48.148896 29218 sys_catalog.cc:343] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:48.148475 29227 sys_catalog.cc:340] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } } I0918 20:39:48.149317 29227 sys_catalog.cc:343] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:48.148957 29103 consensus_queue.cc:229] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } I0918 20:39:48.149938 29218 sys_catalog.cc:428] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:48.151005 29103 sys_catalog.cc:340] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } } I0918 20:39:48.151494 29103 sys_catalog.cc:343] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:48.151321 29228 sys_catalog.cc:340] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } } I0918 20:39:48.152622 29228 sys_catalog.cc:343] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:48.152635 29103 sys_catalog.cc:428] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:48.153892 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 I0918 20:39:48.154119 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 W0918 20:39:48.154724 29229 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:48.154974 29229 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0918 20:39:48.156391 29230 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:48.156724 29230 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:48.170725 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 171592us I0918 20:39:48.171121 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:48.171371 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:48.171582 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:48.171718 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:48.172766 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/ts-0-root/instance: uuid: "a0ce05b791dd4f2ba64117afb154edd9" format_stamp: "Formatted at 2019-09-18 20:39:48 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:48.182716 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.010s sys 0.000s I0918 20:39:48.184381 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:48.192492 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:39:48.192708 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/ts-0-root uuid: "a0ce05b791dd4f2ba64117afb154edd9" format_stamp: "Formatted at 2019-09-18 20:39:48 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:48.192998 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:48.226866 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:48.227030 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:48.266502 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:38113 I0918 20:39:48.268368 19320 webserver.cc:374] Webserver started at http://127.18.222.1:43335/ using document root and password file I0918 20:39:48.289175 29295 heartbeater.cc:322] Connected to a master server at 127.18.222.60:39727 I0918 20:39:48.289582 29295 heartbeater.cc:406] Registering TS with master... I0918 20:39:48.290535 29295 heartbeater.cc:455] Master 127.18.222.60:39727 requested a full tablet report, sending... I0918 20:39:48.293059 29185 ts_manager.cc:186] Registered new tserver with Master: a0ce05b791dd4f2ba64117afb154edd9 (127.18.222.1:38113) I0918 20:39:48.293171 29297 heartbeater.cc:322] Connected to a master server at 127.18.222.62:40629 I0918 20:39:48.293656 29297 heartbeater.cc:406] Registering TS with master... I0918 20:39:48.294226 29297 heartbeater.cc:455] Master 127.18.222.62:40629 requested a full tablet report, sending... I0918 20:39:48.296705 29070 ts_manager.cc:186] Registered new tserver with Master: a0ce05b791dd4f2ba64117afb154edd9 (127.18.222.1:38113) I0918 20:39:48.297422 29296 heartbeater.cc:322] Connected to a master server at 127.18.222.61:39277 I0918 20:39:48.297850 29296 heartbeater.cc:406] Registering TS with master... I0918 20:39:48.298291 29296 heartbeater.cc:455] Master 127.18.222.61:39277 requested a full tablet report, sending... I0918 20:39:48.300249 29127 ts_manager.cc:186] Registered new tserver with Master: a0ce05b791dd4f2ba64117afb154edd9 (127.18.222.1:38113) I0918 20:39:48.303580 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 171592us I0918 20:39:48.303997 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:48.304167 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:48.304388 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:48.304522 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:48.305550 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/ts-1-root/instance: uuid: "bf7c31a99ed64b749aa65aef3a705922" format_stamp: "Formatted at 2019-09-18 20:39:48 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:48.316385 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.011s sys 0.000s I0918 20:39:48.318053 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:48.326970 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:48.327198 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/ts-1-root uuid: "bf7c31a99ed64b749aa65aef3a705922" format_stamp: "Formatted at 2019-09-18 20:39:48 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:48.327453 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:48.362334 29227 raft_consensus.cc:472] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:48.362802 29227 raft_consensus.cc:494] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } I0918 20:39:48.364456 29227 leader_election.cc:296] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38753c8ea6244f958281a39c906cd411 (127.18.222.62:40629), 6e0a29e7c61e40d0a4293b62ec0ac59a (127.18.222.61:39277) I0918 20:39:48.365236 29080 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38753c8ea6244f958281a39c906cd411" is_pre_election: true I0918 20:39:48.366037 29080 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50f4f536877e4dc3b4f97c7d7a214cac in term 0. I0918 20:39:48.365860 29137 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" is_pre_election: true I0918 20:39:48.366541 29137 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50f4f536877e4dc3b4f97c7d7a214cac in term 0. I0918 20:39:48.366976 29167 leader_election.cc:310] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38753c8ea6244f958281a39c906cd411, 50f4f536877e4dc3b4f97c7d7a214cac; no voters: I0918 20:39:48.367942 29227 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:48.368237 29227 raft_consensus.cc:472] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:48.368472 29227 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:48.373592 29227 raft_consensus.cc:494] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } I0918 20:39:48.377125 29080 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38753c8ea6244f958281a39c906cd411" I0918 20:39:48.377681 29080 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:48.377954 29227 leader_election.cc:296] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [CANDIDATE]: Term 1 election: Requested vote from peers 38753c8ea6244f958281a39c906cd411 (127.18.222.62:40629), 6e0a29e7c61e40d0a4293b62ec0ac59a (127.18.222.61:39277) I0918 20:39:48.377467 29137 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" I0918 20:39:48.379978 29137 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:48.386518 29080 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50f4f536877e4dc3b4f97c7d7a214cac in term 1. I0918 20:39:48.388468 29167 leader_election.cc:310] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38753c8ea6244f958281a39c906cd411, 50f4f536877e4dc3b4f97c7d7a214cac; no voters: I0918 20:39:48.389317 29227 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:48.391988 29227 raft_consensus.cc:667] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [term 1 LEADER]: Becoming Leader. State: Replica: 50f4f536877e4dc3b4f97c7d7a214cac, State: Running, Role: LEADER I0918 20:39:48.391885 29137 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50f4f536877e4dc3b4f97c7d7a214cac in term 1. I0918 20:39:48.394407 29227 consensus_queue.cc:206] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } I0918 20:39:48.397989 29308 sys_catalog.cc:340] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50f4f536877e4dc3b4f97c7d7a214cac. Latest consensus state: current_term: 1 leader_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } } I0918 20:39:48.398593 29308 sys_catalog.cc:343] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [sys.catalog]: This master's current role is: LEADER I0918 20:39:48.400707 29313 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:39:48.404242 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:48.404646 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:48.406939 29313 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:39:48.467134 29137 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a [term 1 FOLLOWER]: Refusing update from remote peer 50f4f536877e4dc3b4f97c7d7a214cac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:48.467134 29080 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411 [term 1 FOLLOWER]: Refusing update from remote peer 50f4f536877e4dc3b4f97c7d7a214cac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:48.469058 29308 consensus_queue.cc:984] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:48.469931 29227 consensus_queue.cc:984] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:48.488174 29228 sys_catalog.cc:340] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50f4f536877e4dc3b4f97c7d7a214cac. Latest consensus state: current_term: 1 leader_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } } I0918 20:39:48.489279 29228 sys_catalog.cc:343] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:48.489184 29222 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50f4f536877e4dc3b4f97c7d7a214cac. Latest consensus state: current_term: 1 leader_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } } I0918 20:39:48.490103 29222 sys_catalog.cc:343] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:48.505384 29313 catalog_manager.cc:981] Generated new certificate authority record I0918 20:39:48.509060 29313 catalog_manager.cc:1106] Loading token signing keys... I0918 20:39:48.510838 29227 sys_catalog.cc:340] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } } I0918 20:39:48.511989 29227 sys_catalog.cc:343] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [sys.catalog]: This master's current role is: LEADER I0918 20:39:48.513522 29227 sys_catalog.cc:340] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } } I0918 20:39:48.514350 29227 sys_catalog.cc:343] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [sys.catalog]: This master's current role is: LEADER I0918 20:39:48.516592 29228 sys_catalog.cc:340] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } } I0918 20:39:48.517490 29228 sys_catalog.cc:343] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:48.523375 29222 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38753c8ea6244f958281a39c906cd411" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 40629 } } peers { permanent_uuid: "6e0a29e7c61e40d0a4293b62ec0ac59a" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 39277 } } peers { permanent_uuid: "50f4f536877e4dc3b4f97c7d7a214cac" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 39727 } } } I0918 20:39:48.526502 29222 sys_catalog.cc:343] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:48.549388 29313 catalog_manager.cc:4446] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac: Generated new TSK 0 I0918 20:39:48.550285 29313 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:39:48.564601 29185 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54950 I0918 20:39:48.575953 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.2:43609 I0918 20:39:48.577925 19320 webserver.cc:374] Webserver started at http://127.18.222.2:37961/ using document root and password file I0918 20:39:48.578908 29295 heartbeater.cc:447] Master 127.18.222.60:39727 was elected leader, sending a full tablet report... I0918 20:39:48.614411 29379 heartbeater.cc:322] Connected to a master server at 127.18.222.61:39277 I0918 20:39:48.614786 29379 heartbeater.cc:406] Registering TS with master... I0918 20:39:48.615407 29379 heartbeater.cc:455] Master 127.18.222.61:39277 requested a full tablet report, sending... I0918 20:39:48.618029 29378 heartbeater.cc:322] Connected to a master server at 127.18.222.60:39727 I0918 20:39:48.618353 29378 heartbeater.cc:406] Registering TS with master... I0918 20:39:48.618777 29378 heartbeater.cc:455] Master 127.18.222.60:39727 requested a full tablet report, sending... I0918 20:39:48.620759 29185 ts_manager.cc:186] Registered new tserver with Master: bf7c31a99ed64b749aa65aef3a705922 (127.18.222.2:43609) I0918 20:39:48.620788 29127 ts_manager.cc:186] Registered new tserver with Master: bf7c31a99ed64b749aa65aef3a705922 (127.18.222.2:43609) I0918 20:39:48.622357 29185 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54956 I0918 20:39:48.636519 29378 heartbeater.cc:447] Master 127.18.222.60:39727 was elected leader, sending a full tablet report... I0918 20:39:48.642472 29381 heartbeater.cc:322] Connected to a master server at 127.18.222.62:40629 I0918 20:39:48.642807 29381 heartbeater.cc:406] Registering TS with master... I0918 20:39:48.643388 29381 heartbeater.cc:455] Master 127.18.222.62:40629 requested a full tablet report, sending... I0918 20:39:48.643832 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 171592us I0918 20:39:48.644227 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:48.644402 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:48.645609 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:48.645885 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:48.646071 29070 ts_manager.cc:186] Registered new tserver with Master: bf7c31a99ed64b749aa65aef3a705922 (127.18.222.2:43609) I0918 20:39:48.647441 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/ts-2-root/instance: uuid: "e83f0943ebc44081a9df5fc862483865" format_stamp: "Formatted at 2019-09-18 20:39:48 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:48.664284 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.016s user 0.015s sys 0.001s I0918 20:39:48.666702 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:48.679340 19320 fs_manager.cc:419] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s I0918 20:39:48.679587 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/ts-2-root uuid: "e83f0943ebc44081a9df5fc862483865" format_stamp: "Formatted at 2019-09-18 20:39:48 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:48.679893 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:48.807188 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:48.807425 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:48.862314 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.3:37065 I0918 20:39:48.864243 19320 webserver.cc:374] Webserver started at http://127.18.222.3:36029/ using document root and password file I0918 20:39:48.902581 29451 heartbeater.cc:322] Connected to a master server at 127.18.222.62:40629 I0918 20:39:48.903090 29451 heartbeater.cc:406] Registering TS with master... I0918 20:39:48.903795 29451 heartbeater.cc:455] Master 127.18.222.62:40629 requested a full tablet report, sending... I0918 20:39:48.906402 29069 ts_manager.cc:186] Registered new tserver with Master: e83f0943ebc44081a9df5fc862483865 (127.18.222.3:37065) I0918 20:39:48.914269 29450 heartbeater.cc:322] Connected to a master server at 127.18.222.61:39277 I0918 20:39:48.914392 29449 heartbeater.cc:322] Connected to a master server at 127.18.222.60:39727 I0918 20:39:48.914906 29449 heartbeater.cc:406] Registering TS with master... I0918 20:39:48.915469 29449 heartbeater.cc:455] Master 127.18.222.60:39727 requested a full tablet report, sending... I0918 20:39:48.916201 29450 heartbeater.cc:406] Registering TS with master... I0918 20:39:48.916788 29450 heartbeater.cc:455] Master 127.18.222.61:39277 requested a full tablet report, sending... I0918 20:39:48.917641 29184 ts_manager.cc:186] Registered new tserver with Master: e83f0943ebc44081a9df5fc862483865 (127.18.222.3:37065) I0918 20:39:48.919127 29126 ts_manager.cc:186] Registered new tserver with Master: e83f0943ebc44081a9df5fc862483865 (127.18.222.3:37065) I0918 20:39:48.920222 29184 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54962 I0918 20:39:48.924058 19320 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.05472684s I0918 20:39:48.934001 29449 heartbeater.cc:447] Master 127.18.222.60:39727 was elected leader, sending a full tablet report... I0918 20:39:48.983847 29184 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54972: name: "ksck-test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\n\000\000\000\004\001\024\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:39:48.985893 29184 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:39:49.100270 29262 tablet_service.cc:1022] Processing CreateTablet for tablet 7d73db44d1c64fab8cba98a77804d6c3 (table=ksck-test-table [id=b179c877fa324d70b0351ea40ea5096d]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:49.100792 29261 tablet_service.cc:1022] Processing CreateTablet for tablet c60b25182baf4c98b26d70efebd27734 (table=ksck-test-table [id=b179c877fa324d70b0351ea40ea5096d]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:49.101390 29262 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 7d73db44d1c64fab8cba98a77804d6c3. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:49.101845 29261 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet c60b25182baf4c98b26d70efebd27734. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:49.104293 29260 tablet_service.cc:1022] Processing CreateTablet for tablet d3484a1195b54f0c87d20ed7b6c38c32 (table=ksck-test-table [id=b179c877fa324d70b0351ea40ea5096d]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:49.105358 29260 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet d3484a1195b54f0c87d20ed7b6c38c32. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:49.126824 29336 tablet_service.cc:1022] Processing CreateTablet for tablet c60b25182baf4c98b26d70efebd27734 (table=ksck-test-table [id=b179c877fa324d70b0351ea40ea5096d]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:49.127090 29337 tablet_service.cc:1022] Processing CreateTablet for tablet 7d73db44d1c64fab8cba98a77804d6c3 (table=ksck-test-table [id=b179c877fa324d70b0351ea40ea5096d]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:49.128520 29336 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet c60b25182baf4c98b26d70efebd27734. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:49.128327 29335 tablet_service.cc:1022] Processing CreateTablet for tablet d3484a1195b54f0c87d20ed7b6c38c32 (table=ksck-test-table [id=b179c877fa324d70b0351ea40ea5096d]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:49.129362 29335 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet d3484a1195b54f0c87d20ed7b6c38c32. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:49.132227 29337 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 7d73db44d1c64fab8cba98a77804d6c3. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:49.132962 29416 tablet_service.cc:1022] Processing CreateTablet for tablet 7d73db44d1c64fab8cba98a77804d6c3 (table=ksck-test-table [id=b179c877fa324d70b0351ea40ea5096d]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:49.133433 29415 tablet_service.cc:1022] Processing CreateTablet for tablet c60b25182baf4c98b26d70efebd27734 (table=ksck-test-table [id=b179c877fa324d70b0351ea40ea5096d]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:49.134038 29416 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 7d73db44d1c64fab8cba98a77804d6c3. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:49.134568 29415 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet c60b25182baf4c98b26d70efebd27734. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:49.167165 29414 tablet_service.cc:1022] Processing CreateTablet for tablet d3484a1195b54f0c87d20ed7b6c38c32 (table=ksck-test-table [id=b179c877fa324d70b0351ea40ea5096d]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:49.168522 29414 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet d3484a1195b54f0c87d20ed7b6c38c32. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:49.194941 29223 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a: acquiring CA information for follower catalog manager: success I0918 20:39:49.197827 29471 tablet_bootstrap.cc:439] T c60b25182baf4c98b26d70efebd27734 P e83f0943ebc44081a9df5fc862483865: Bootstrap starting. I0918 20:39:49.203001 29471 tablet_bootstrap.cc:590] T c60b25182baf4c98b26d70efebd27734 P e83f0943ebc44081a9df5fc862483865: No blocks or log segments found. Creating new log. I0918 20:39:49.199441 29230 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411: acquiring CA information for follower catalog manager: success I0918 20:39:49.217900 29470 tablet_bootstrap.cc:439] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9: Bootstrap starting. I0918 20:39:49.222384 29470 tablet_bootstrap.cc:590] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9: No blocks or log segments found. Creating new log. I0918 20:39:49.238041 29230 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:49.239279 29472 tablet_bootstrap.cc:439] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922: Bootstrap starting. I0918 20:39:49.239471 29223 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:49.244202 29472 tablet_bootstrap.cc:590] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922: No blocks or log segments found. Creating new log. I0918 20:39:49.244143 29471 tablet_bootstrap.cc:439] T c60b25182baf4c98b26d70efebd27734 P e83f0943ebc44081a9df5fc862483865: No bootstrap required, opened a new log I0918 20:39:49.257570 29471 ts_tablet_manager.cc:1085] T c60b25182baf4c98b26d70efebd27734 P e83f0943ebc44081a9df5fc862483865: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.001s I0918 20:39:49.259497 29471 raft_consensus.cc:340] T c60b25182baf4c98b26d70efebd27734 P e83f0943ebc44081a9df5fc862483865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:49.260257 29471 raft_consensus.cc:366] T c60b25182baf4c98b26d70efebd27734 P e83f0943ebc44081a9df5fc862483865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:49.260499 29471 raft_consensus.cc:705] T c60b25182baf4c98b26d70efebd27734 P e83f0943ebc44081a9df5fc862483865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e83f0943ebc44081a9df5fc862483865, State: Initialized, Role: FOLLOWER I0918 20:39:49.273625 29472 tablet_bootstrap.cc:439] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922: No bootstrap required, opened a new log I0918 20:39:49.274253 29472 ts_tablet_manager.cc:1085] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922: Time spent bootstrapping tablet: real 0.035s user 0.018s sys 0.015s I0918 20:39:49.280745 29470 tablet_bootstrap.cc:439] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9: No bootstrap required, opened a new log I0918 20:39:49.282106 29471 consensus_queue.cc:229] T c60b25182baf4c98b26d70efebd27734 P e83f0943ebc44081a9df5fc862483865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:49.284163 29470 ts_tablet_manager.cc:1085] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9: Time spent bootstrapping tablet: real 0.073s user 0.008s sys 0.000s I0918 20:39:49.280344 29472 raft_consensus.cc:340] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } I0918 20:39:49.287783 29472 raft_consensus.cc:366] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:49.287999 29472 raft_consensus.cc:705] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7c31a99ed64b749aa65aef3a705922, State: Initialized, Role: FOLLOWER I0918 20:39:49.288581 29472 consensus_queue.cc:229] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } I0918 20:39:49.289775 29470 raft_consensus.cc:340] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:49.290838 29470 raft_consensus.cc:366] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:49.291496 29470 raft_consensus.cc:705] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0ce05b791dd4f2ba64117afb154edd9, State: Initialized, Role: FOLLOWER I0918 20:39:49.292534 29470 consensus_queue.cc:229] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:49.302871 29471 ts_tablet_manager.cc:1113] T c60b25182baf4c98b26d70efebd27734 P e83f0943ebc44081a9df5fc862483865: Time spent starting tablet: real 0.045s user 0.003s sys 0.014s I0918 20:39:49.303994 29471 tablet_bootstrap.cc:439] T d3484a1195b54f0c87d20ed7b6c38c32 P e83f0943ebc44081a9df5fc862483865: Bootstrap starting. I0918 20:39:49.307626 29471 tablet_bootstrap.cc:590] T d3484a1195b54f0c87d20ed7b6c38c32 P e83f0943ebc44081a9df5fc862483865: No blocks or log segments found. Creating new log. I0918 20:39:49.335443 29472 ts_tablet_manager.cc:1113] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922: Time spent starting tablet: real 0.061s user 0.007s sys 0.000s I0918 20:39:49.345150 29471 tablet_bootstrap.cc:439] T d3484a1195b54f0c87d20ed7b6c38c32 P e83f0943ebc44081a9df5fc862483865: No bootstrap required, opened a new log I0918 20:39:49.345904 29471 ts_tablet_manager.cc:1085] T d3484a1195b54f0c87d20ed7b6c38c32 P e83f0943ebc44081a9df5fc862483865: Time spent bootstrapping tablet: real 0.042s user 0.017s sys 0.005s I0918 20:39:49.349298 29470 ts_tablet_manager.cc:1113] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9: Time spent starting tablet: real 0.062s user 0.004s sys 0.003s I0918 20:39:49.350256 29470 tablet_bootstrap.cc:439] T d3484a1195b54f0c87d20ed7b6c38c32 P a0ce05b791dd4f2ba64117afb154edd9: Bootstrap starting. I0918 20:39:49.348088 29471 raft_consensus.cc:340] T d3484a1195b54f0c87d20ed7b6c38c32 P e83f0943ebc44081a9df5fc862483865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:49.352396 29471 raft_consensus.cc:366] T d3484a1195b54f0c87d20ed7b6c38c32 P e83f0943ebc44081a9df5fc862483865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:49.352735 29471 raft_consensus.cc:705] T d3484a1195b54f0c87d20ed7b6c38c32 P e83f0943ebc44081a9df5fc862483865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e83f0943ebc44081a9df5fc862483865, State: Initialized, Role: FOLLOWER I0918 20:39:49.353979 29470 tablet_bootstrap.cc:590] T d3484a1195b54f0c87d20ed7b6c38c32 P a0ce05b791dd4f2ba64117afb154edd9: No blocks or log segments found. Creating new log. I0918 20:39:49.353581 29471 consensus_queue.cc:229] T d3484a1195b54f0c87d20ed7b6c38c32 P e83f0943ebc44081a9df5fc862483865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:49.346012 29472 tablet_bootstrap.cc:439] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922: Bootstrap starting. W0918 20:39:49.357874 29299 tablet.cc:1763] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9: Can't schedule compaction. Clean time has not been advanced past its initial value. I0918 20:39:49.360700 29472 tablet_bootstrap.cc:590] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922: No blocks or log segments found. Creating new log. I0918 20:39:49.369487 29471 ts_tablet_manager.cc:1113] T d3484a1195b54f0c87d20ed7b6c38c32 P e83f0943ebc44081a9df5fc862483865: Time spent starting tablet: real 0.023s user 0.006s sys 0.001s I0918 20:39:49.374522 29471 tablet_bootstrap.cc:439] T 7d73db44d1c64fab8cba98a77804d6c3 P e83f0943ebc44081a9df5fc862483865: Bootstrap starting. I0918 20:39:49.371922 29470 tablet_bootstrap.cc:439] T d3484a1195b54f0c87d20ed7b6c38c32 P a0ce05b791dd4f2ba64117afb154edd9: No bootstrap required, opened a new log I0918 20:39:49.376160 29470 ts_tablet_manager.cc:1085] T d3484a1195b54f0c87d20ed7b6c38c32 P a0ce05b791dd4f2ba64117afb154edd9: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.006s I0918 20:39:49.381407 29471 tablet_bootstrap.cc:590] T 7d73db44d1c64fab8cba98a77804d6c3 P e83f0943ebc44081a9df5fc862483865: No blocks or log segments found. Creating new log. I0918 20:39:49.381546 29479 raft_consensus.cc:472] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:49.378445 29470 raft_consensus.cc:340] T d3484a1195b54f0c87d20ed7b6c38c32 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:49.382644 29470 raft_consensus.cc:366] T d3484a1195b54f0c87d20ed7b6c38c32 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:49.382210 29479 raft_consensus.cc:494] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:49.382918 29470 raft_consensus.cc:705] T d3484a1195b54f0c87d20ed7b6c38c32 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0ce05b791dd4f2ba64117afb154edd9, State: Initialized, Role: FOLLOWER I0918 20:39:49.385432 29472 tablet_bootstrap.cc:439] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922: No bootstrap required, opened a new log I0918 20:39:49.386034 29472 ts_tablet_manager.cc:1085] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.000s I0918 20:39:49.388784 29470 consensus_queue.cc:229] T d3484a1195b54f0c87d20ed7b6c38c32 P a0ce05b791dd4f2ba64117afb154edd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:49.391239 29479 leader_election.cc:296] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf7c31a99ed64b749aa65aef3a705922 (127.18.222.2:43609), e83f0943ebc44081a9df5fc862483865 (127.18.222.3:37065) I0918 20:39:49.407352 29471 tablet_bootstrap.cc:439] T 7d73db44d1c64fab8cba98a77804d6c3 P e83f0943ebc44081a9df5fc862483865: No bootstrap required, opened a new log I0918 20:39:49.409025 29471 ts_tablet_manager.cc:1085] T 7d73db44d1c64fab8cba98a77804d6c3 P e83f0943ebc44081a9df5fc862483865: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.001s I0918 20:39:49.411861 29471 raft_consensus.cc:340] T 7d73db44d1c64fab8cba98a77804d6c3 P e83f0943ebc44081a9df5fc862483865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } I0918 20:39:49.412678 29471 raft_consensus.cc:366] T 7d73db44d1c64fab8cba98a77804d6c3 P e83f0943ebc44081a9df5fc862483865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:49.412942 29471 raft_consensus.cc:705] T 7d73db44d1c64fab8cba98a77804d6c3 P e83f0943ebc44081a9df5fc862483865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e83f0943ebc44081a9df5fc862483865, State: Initialized, Role: FOLLOWER I0918 20:39:49.388257 29472 raft_consensus.cc:340] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:49.423347 29472 raft_consensus.cc:366] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:49.424403 29472 raft_consensus.cc:705] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7c31a99ed64b749aa65aef3a705922, State: Initialized, Role: FOLLOWER I0918 20:39:49.427862 29477 raft_consensus.cc:472] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:49.427409 29472 consensus_queue.cc:229] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:49.428432 29477 raft_consensus.cc:494] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } I0918 20:39:49.430660 29477 leader_election.cc:296] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e83f0943ebc44081a9df5fc862483865 (127.18.222.3:37065), a0ce05b791dd4f2ba64117afb154edd9 (127.18.222.1:38113) I0918 20:39:49.429342 29354 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c60b25182baf4c98b26d70efebd27734" candidate_uuid: "a0ce05b791dd4f2ba64117afb154edd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf7c31a99ed64b749aa65aef3a705922" is_pre_election: true I0918 20:39:49.430343 29470 ts_tablet_manager.cc:1113] T d3484a1195b54f0c87d20ed7b6c38c32 P a0ce05b791dd4f2ba64117afb154edd9: Time spent starting tablet: real 0.054s user 0.018s sys 0.003s I0918 20:39:49.472380 29470 tablet_bootstrap.cc:439] T 7d73db44d1c64fab8cba98a77804d6c3 P a0ce05b791dd4f2ba64117afb154edd9: Bootstrap starting. I0918 20:39:49.485083 29470 tablet_bootstrap.cc:590] T 7d73db44d1c64fab8cba98a77804d6c3 P a0ce05b791dd4f2ba64117afb154edd9: No blocks or log segments found. Creating new log. I0918 20:39:49.427181 29471 consensus_queue.cc:229] T 7d73db44d1c64fab8cba98a77804d6c3 P e83f0943ebc44081a9df5fc862483865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } I0918 20:39:49.488087 29471 ts_tablet_manager.cc:1113] T 7d73db44d1c64fab8cba98a77804d6c3 P e83f0943ebc44081a9df5fc862483865: Time spent starting tablet: real 0.079s user 0.007s sys 0.001s W0918 20:39:49.501201 29236 leader_election.cc:348] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf7c31a99ed64b749aa65aef3a705922 (127.18.222.2:43609): Illegal state: must be running to vote when last-logged opid is not known I0918 20:39:49.535950 29472 ts_tablet_manager.cc:1113] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922: Time spent starting tablet: real 0.149s user 0.013s sys 0.024s I0918 20:39:49.544709 29472 tablet_bootstrap.cc:439] T c60b25182baf4c98b26d70efebd27734 P bf7c31a99ed64b749aa65aef3a705922: Bootstrap starting. I0918 20:39:49.555387 29426 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c60b25182baf4c98b26d70efebd27734" candidate_uuid: "a0ce05b791dd4f2ba64117afb154edd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e83f0943ebc44081a9df5fc862483865" is_pre_election: true I0918 20:39:49.557637 29426 raft_consensus.cc:2350] T c60b25182baf4c98b26d70efebd27734 P e83f0943ebc44081a9df5fc862483865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0ce05b791dd4f2ba64117afb154edd9 in term 0. I0918 20:39:49.561902 29470 tablet_bootstrap.cc:439] T 7d73db44d1c64fab8cba98a77804d6c3 P a0ce05b791dd4f2ba64117afb154edd9: No bootstrap required, opened a new log I0918 20:39:49.569144 29234 leader_election.cc:310] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0ce05b791dd4f2ba64117afb154edd9, e83f0943ebc44081a9df5fc862483865; no voters: bf7c31a99ed64b749aa65aef3a705922 I0918 20:39:49.572891 29484 raft_consensus.cc:2644] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:49.573220 29484 raft_consensus.cc:472] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:49.573513 29484 raft_consensus.cc:2890] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:49.573896 29472 tablet_bootstrap.cc:590] T c60b25182baf4c98b26d70efebd27734 P bf7c31a99ed64b749aa65aef3a705922: No blocks or log segments found. Creating new log. I0918 20:39:49.577960 29472 tablet_bootstrap.cc:439] T c60b25182baf4c98b26d70efebd27734 P bf7c31a99ed64b749aa65aef3a705922: No bootstrap required, opened a new log I0918 20:39:49.578536 29472 ts_tablet_manager.cc:1085] T c60b25182baf4c98b26d70efebd27734 P bf7c31a99ed64b749aa65aef3a705922: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.001s I0918 20:39:49.583415 29272 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7d73db44d1c64fab8cba98a77804d6c3" candidate_uuid: "bf7c31a99ed64b749aa65aef3a705922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0ce05b791dd4f2ba64117afb154edd9" is_pre_election: true W0918 20:39:49.585976 29304 leader_election.cc:348] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0ce05b791dd4f2ba64117afb154edd9 (127.18.222.1:38113): Illegal state: must be running to vote when last-logged opid is not known I0918 20:39:49.585386 29472 raft_consensus.cc:340] T c60b25182baf4c98b26d70efebd27734 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:49.586526 29472 raft_consensus.cc:366] T c60b25182baf4c98b26d70efebd27734 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:49.586877 29472 raft_consensus.cc:705] T c60b25182baf4c98b26d70efebd27734 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7c31a99ed64b749aa65aef3a705922, State: Initialized, Role: FOLLOWER I0918 20:39:49.587654 29472 consensus_queue.cc:229] T c60b25182baf4c98b26d70efebd27734 P bf7c31a99ed64b749aa65aef3a705922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:49.589054 29484 raft_consensus.cc:494] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:49.591799 29426 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c60b25182baf4c98b26d70efebd27734" candidate_uuid: "a0ce05b791dd4f2ba64117afb154edd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e83f0943ebc44081a9df5fc862483865" I0918 20:39:49.592394 29426 raft_consensus.cc:2890] T c60b25182baf4c98b26d70efebd27734 P e83f0943ebc44081a9df5fc862483865 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:49.595003 29472 ts_tablet_manager.cc:1113] T c60b25182baf4c98b26d70efebd27734 P bf7c31a99ed64b749aa65aef3a705922: Time spent starting tablet: real 0.016s user 0.006s sys 0.002s I0918 20:39:49.600188 29425 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7d73db44d1c64fab8cba98a77804d6c3" candidate_uuid: "bf7c31a99ed64b749aa65aef3a705922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e83f0943ebc44081a9df5fc862483865" is_pre_election: true I0918 20:39:49.600869 29425 raft_consensus.cc:2350] T 7d73db44d1c64fab8cba98a77804d6c3 P e83f0943ebc44081a9df5fc862483865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf7c31a99ed64b749aa65aef3a705922 in term 0. I0918 20:39:49.602051 29305 leader_election.cc:310] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf7c31a99ed64b749aa65aef3a705922, e83f0943ebc44081a9df5fc862483865; no voters: a0ce05b791dd4f2ba64117afb154edd9 I0918 20:39:49.602846 29486 raft_consensus.cc:2644] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:49.603135 29486 raft_consensus.cc:472] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:49.603356 29486 raft_consensus.cc:2890] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:49.605162 29426 raft_consensus.cc:2350] T c60b25182baf4c98b26d70efebd27734 P e83f0943ebc44081a9df5fc862483865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0ce05b791dd4f2ba64117afb154edd9 in term 1. I0918 20:39:49.606132 29470 ts_tablet_manager.cc:1085] T 7d73db44d1c64fab8cba98a77804d6c3 P a0ce05b791dd4f2ba64117afb154edd9: Time spent bootstrapping tablet: real 0.134s user 0.013s sys 0.005s I0918 20:39:49.606201 29234 leader_election.cc:310] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0ce05b791dd4f2ba64117afb154edd9, e83f0943ebc44081a9df5fc862483865; no voters: I0918 20:39:49.607192 29479 raft_consensus.cc:2644] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:49.608111 29354 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c60b25182baf4c98b26d70efebd27734" candidate_uuid: "a0ce05b791dd4f2ba64117afb154edd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf7c31a99ed64b749aa65aef3a705922" I0918 20:39:49.608845 29354 raft_consensus.cc:2890] T c60b25182baf4c98b26d70efebd27734 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:49.608397 29486 raft_consensus.cc:494] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } I0918 20:39:49.610425 29484 leader_election.cc:296] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [CANDIDATE]: Term 1 election: Requested vote from peers bf7c31a99ed64b749aa65aef3a705922 (127.18.222.2:43609), e83f0943ebc44081a9df5fc862483865 (127.18.222.3:37065) I0918 20:39:49.613229 29272 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7d73db44d1c64fab8cba98a77804d6c3" candidate_uuid: "bf7c31a99ed64b749aa65aef3a705922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0ce05b791dd4f2ba64117afb154edd9" W0918 20:39:49.619124 29304 leader_election.cc:348] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0ce05b791dd4f2ba64117afb154edd9 (127.18.222.1:38113): Illegal state: must be running to vote when last-logged opid is not known I0918 20:39:49.620328 29486 leader_election.cc:296] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [CANDIDATE]: Term 1 election: Requested vote from peers e83f0943ebc44081a9df5fc862483865 (127.18.222.3:37065), a0ce05b791dd4f2ba64117afb154edd9 (127.18.222.1:38113) I0918 20:39:49.624092 29426 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7d73db44d1c64fab8cba98a77804d6c3" candidate_uuid: "bf7c31a99ed64b749aa65aef3a705922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e83f0943ebc44081a9df5fc862483865" I0918 20:39:49.623425 29470 raft_consensus.cc:340] T 7d73db44d1c64fab8cba98a77804d6c3 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } I0918 20:39:49.626320 29470 raft_consensus.cc:366] T 7d73db44d1c64fab8cba98a77804d6c3 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:49.626713 29470 raft_consensus.cc:705] T 7d73db44d1c64fab8cba98a77804d6c3 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0ce05b791dd4f2ba64117afb154edd9, State: Initialized, Role: FOLLOWER I0918 20:39:49.628432 29470 consensus_queue.cc:229] T 7d73db44d1c64fab8cba98a77804d6c3 P a0ce05b791dd4f2ba64117afb154edd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } I0918 20:39:49.635069 29426 raft_consensus.cc:2890] T 7d73db44d1c64fab8cba98a77804d6c3 P e83f0943ebc44081a9df5fc862483865 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:49.636126 29470 ts_tablet_manager.cc:1113] T 7d73db44d1c64fab8cba98a77804d6c3 P a0ce05b791dd4f2ba64117afb154edd9: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s I0918 20:39:49.637481 29479 raft_consensus.cc:667] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [term 1 LEADER]: Becoming Leader. State: Replica: a0ce05b791dd4f2ba64117afb154edd9, State: Running, Role: LEADER I0918 20:39:49.647521 29426 raft_consensus.cc:2350] T 7d73db44d1c64fab8cba98a77804d6c3 P e83f0943ebc44081a9df5fc862483865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf7c31a99ed64b749aa65aef3a705922 in term 1. I0918 20:39:49.638550 29479 consensus_queue.cc:206] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:49.650125 29305 leader_election.cc:310] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf7c31a99ed64b749aa65aef3a705922, e83f0943ebc44081a9df5fc862483865; no voters: a0ce05b791dd4f2ba64117afb154edd9 I0918 20:39:49.651161 29486 raft_consensus.cc:2644] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:49.651685 29486 raft_consensus.cc:667] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [term 1 LEADER]: Becoming Leader. State: Replica: bf7c31a99ed64b749aa65aef3a705922, State: Running, Role: LEADER I0918 20:39:49.652393 29486 consensus_queue.cc:206] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } I0918 20:39:49.655658 29486 raft_consensus.cc:472] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:49.656113 29486 raft_consensus.cc:494] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:49.657316 29183 catalog_manager.cc:4131] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 reported cstate change: term changed from 0 to 1, leader changed from to a0ce05b791dd4f2ba64117afb154edd9 (127.18.222.1). New cstate: current_term: 1 leader_uuid: "a0ce05b791dd4f2ba64117afb154edd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } health_report { overall_health: UNKNOWN } } } I0918 20:39:49.658531 29272 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d3484a1195b54f0c87d20ed7b6c38c32" candidate_uuid: "bf7c31a99ed64b749aa65aef3a705922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0ce05b791dd4f2ba64117afb154edd9" is_pre_election: true I0918 20:39:49.659204 29272 raft_consensus.cc:2350] T d3484a1195b54f0c87d20ed7b6c38c32 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf7c31a99ed64b749aa65aef3a705922 in term 0. I0918 20:39:49.660185 29304 leader_election.cc:310] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0ce05b791dd4f2ba64117afb154edd9, bf7c31a99ed64b749aa65aef3a705922; no voters: I0918 20:39:49.661084 29477 raft_consensus.cc:2644] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:49.661411 29477 raft_consensus.cc:472] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:49.661904 29477 raft_consensus.cc:2890] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:49.661343 29184 catalog_manager.cc:4131] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 reported cstate change: term changed from 0 to 1, leader changed from to bf7c31a99ed64b749aa65aef3a705922 (127.18.222.2). New cstate: current_term: 1 leader_uuid: "bf7c31a99ed64b749aa65aef3a705922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } health_report { overall_health: HEALTHY } } } I0918 20:39:49.668203 29477 raft_consensus.cc:494] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:49.647402 29354 raft_consensus.cc:2350] T c60b25182baf4c98b26d70efebd27734 P bf7c31a99ed64b749aa65aef3a705922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0ce05b791dd4f2ba64117afb154edd9 in term 1. I0918 20:39:49.672204 29272 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d3484a1195b54f0c87d20ed7b6c38c32" candidate_uuid: "bf7c31a99ed64b749aa65aef3a705922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0ce05b791dd4f2ba64117afb154edd9" I0918 20:39:49.673666 29486 leader_election.cc:296] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0ce05b791dd4f2ba64117afb154edd9 (127.18.222.1:38113), e83f0943ebc44081a9df5fc862483865 (127.18.222.3:37065) I0918 20:39:49.674038 29272 raft_consensus.cc:2890] T d3484a1195b54f0c87d20ed7b6c38c32 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:49.675231 29477 leader_election.cc:296] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 [CANDIDATE]: Term 1 election: Requested vote from peers a0ce05b791dd4f2ba64117afb154edd9 (127.18.222.1:38113), e83f0943ebc44081a9df5fc862483865 (127.18.222.3:37065) I0918 20:39:49.683007 29272 raft_consensus.cc:2350] T d3484a1195b54f0c87d20ed7b6c38c32 P a0ce05b791dd4f2ba64117afb154edd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf7c31a99ed64b749aa65aef3a705922 in term 1. I0918 20:39:49.683537 29425 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d3484a1195b54f0c87d20ed7b6c38c32" candidate_uuid: "bf7c31a99ed64b749aa65aef3a705922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e83f0943ebc44081a9df5fc862483865" I0918 20:39:49.684792 29425 raft_consensus.cc:2890] T d3484a1195b54f0c87d20ed7b6c38c32 P e83f0943ebc44081a9df5fc862483865 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:49.689195 29304 leader_election.cc:310] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0ce05b791dd4f2ba64117afb154edd9, bf7c31a99ed64b749aa65aef3a705922; no voters: I0918 20:39:49.690305 29477 raft_consensus.cc:2644] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:49.690750 29477 raft_consensus.cc:667] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 [term 1 LEADER]: Becoming Leader. State: Replica: bf7c31a99ed64b749aa65aef3a705922, State: Running, Role: LEADER I0918 20:39:49.683164 29426 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d3484a1195b54f0c87d20ed7b6c38c32" candidate_uuid: "bf7c31a99ed64b749aa65aef3a705922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e83f0943ebc44081a9df5fc862483865" is_pre_election: true I0918 20:39:49.695783 29425 raft_consensus.cc:2350] T d3484a1195b54f0c87d20ed7b6c38c32 P e83f0943ebc44081a9df5fc862483865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf7c31a99ed64b749aa65aef3a705922 in term 1. I0918 20:39:49.697518 29426 raft_consensus.cc:2325] T d3484a1195b54f0c87d20ed7b6c38c32 P e83f0943ebc44081a9df5fc862483865 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bf7c31a99ed64b749aa65aef3a705922 for term 1 because replica is already servicing an update from a current leader or another vote. I0918 20:39:49.691357 29477 consensus_queue.cc:206] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:49.720891 29479 raft_consensus.cc:472] T 7d73db44d1c64fab8cba98a77804d6c3 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:49.721666 29479 raft_consensus.cc:494] T 7d73db44d1c64fab8cba98a77804d6c3 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } I0918 20:39:49.724825 29354 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7d73db44d1c64fab8cba98a77804d6c3" candidate_uuid: "a0ce05b791dd4f2ba64117afb154edd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf7c31a99ed64b749aa65aef3a705922" is_pre_election: true I0918 20:39:49.725586 29354 raft_consensus.cc:2311] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a0ce05b791dd4f2ba64117afb154edd9 for term 1 because replica is either leader or believes a valid leader to be alive. I0918 20:39:49.725847 29425 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7d73db44d1c64fab8cba98a77804d6c3" candidate_uuid: "a0ce05b791dd4f2ba64117afb154edd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e83f0943ebc44081a9df5fc862483865" is_pre_election: true I0918 20:39:49.726090 29479 leader_election.cc:296] T 7d73db44d1c64fab8cba98a77804d6c3 P a0ce05b791dd4f2ba64117afb154edd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e83f0943ebc44081a9df5fc862483865 (127.18.222.3:37065), bf7c31a99ed64b749aa65aef3a705922 (127.18.222.2:43609) I0918 20:39:49.728121 29425 raft_consensus.cc:2280] T 7d73db44d1c64fab8cba98a77804d6c3 P e83f0943ebc44081a9df5fc862483865 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0ce05b791dd4f2ba64117afb154edd9 in current term 1: Already voted for candidate bf7c31a99ed64b749aa65aef3a705922 in this term. I0918 20:39:49.730651 29234 leader_election.cc:310] T 7d73db44d1c64fab8cba98a77804d6c3 P a0ce05b791dd4f2ba64117afb154edd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0ce05b791dd4f2ba64117afb154edd9; no voters: bf7c31a99ed64b749aa65aef3a705922, e83f0943ebc44081a9df5fc862483865 I0918 20:39:49.732482 29479 raft_consensus.cc:2890] T 7d73db44d1c64fab8cba98a77804d6c3 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:49.741698 29479 raft_consensus.cc:2592] T 7d73db44d1c64fab8cba98a77804d6c3 P a0ce05b791dd4f2ba64117afb154edd9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W0918 20:39:49.756381 29170 debug-util.cc:397] Leaking SignalData structure 0x7b0800126a00 after lost signal to thread 19323 I0918 20:39:49.762151 29184 catalog_manager.cc:4131] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 reported cstate change: term changed from 0 to 1, leader changed from to bf7c31a99ed64b749aa65aef3a705922 (127.18.222.2). New cstate: current_term: 1 leader_uuid: "bf7c31a99ed64b749aa65aef3a705922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } health_report { overall_health: UNKNOWN } } } I0918 20:39:49.906882 29182 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54972: name: "other-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "value" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\006\001\000\000\000\000\007\001d\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:39:49.908684 29182 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table other-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:39:49.966156 29262 tablet_service.cc:1022] Processing CreateTablet for tablet 413283585b6d41d5b6c0a0d032a86c37 (table=other-table [id=a65a618babc844b18b588a6fde653e02]), partition=RANGE (key) PARTITION 0 <= VALUES < 100 I0918 20:39:49.967206 29262 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 413283585b6d41d5b6c0a0d032a86c37. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:49.975296 29337 tablet_service.cc:1022] Processing CreateTablet for tablet 413283585b6d41d5b6c0a0d032a86c37 (table=other-table [id=a65a618babc844b18b588a6fde653e02]), partition=RANGE (key) PARTITION 0 <= VALUES < 100 I0918 20:39:49.976598 29337 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 413283585b6d41d5b6c0a0d032a86c37. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:49.988888 29416 tablet_service.cc:1022] Processing CreateTablet for tablet 413283585b6d41d5b6c0a0d032a86c37 (table=other-table [id=a65a618babc844b18b588a6fde653e02]), partition=RANGE (key) PARTITION 0 <= VALUES < 100 I0918 20:39:49.992416 29416 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 413283585b6d41d5b6c0a0d032a86c37. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:49.999423 29470 tablet_bootstrap.cc:439] T 413283585b6d41d5b6c0a0d032a86c37 P a0ce05b791dd4f2ba64117afb154edd9: Bootstrap starting. I0918 20:39:50.004520 29470 tablet_bootstrap.cc:590] T 413283585b6d41d5b6c0a0d032a86c37 P a0ce05b791dd4f2ba64117afb154edd9: No blocks or log segments found. Creating new log. I0918 20:39:50.012132 29472 tablet_bootstrap.cc:439] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922: Bootstrap starting. I0918 20:39:50.016710 29472 tablet_bootstrap.cc:590] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922: No blocks or log segments found. Creating new log. I0918 20:39:50.022989 29470 tablet_bootstrap.cc:439] T 413283585b6d41d5b6c0a0d032a86c37 P a0ce05b791dd4f2ba64117afb154edd9: No bootstrap required, opened a new log I0918 20:39:50.023743 29470 ts_tablet_manager.cc:1085] T 413283585b6d41d5b6c0a0d032a86c37 P a0ce05b791dd4f2ba64117afb154edd9: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.000s I0918 20:39:50.029248 29470 raft_consensus.cc:340] T 413283585b6d41d5b6c0a0d032a86c37 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:50.030105 29470 raft_consensus.cc:366] T 413283585b6d41d5b6c0a0d032a86c37 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:50.030337 29470 raft_consensus.cc:705] T 413283585b6d41d5b6c0a0d032a86c37 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0ce05b791dd4f2ba64117afb154edd9, State: Initialized, Role: FOLLOWER I0918 20:39:50.031141 29470 consensus_queue.cc:229] T 413283585b6d41d5b6c0a0d032a86c37 P a0ce05b791dd4f2ba64117afb154edd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:50.034260 29510 tablet_bootstrap.cc:439] T 413283585b6d41d5b6c0a0d032a86c37 P e83f0943ebc44081a9df5fc862483865: Bootstrap starting. I0918 20:39:50.039613 29510 tablet_bootstrap.cc:590] T 413283585b6d41d5b6c0a0d032a86c37 P e83f0943ebc44081a9df5fc862483865: No blocks or log segments found. Creating new log. I0918 20:39:50.043787 29472 tablet_bootstrap.cc:439] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922: No bootstrap required, opened a new log I0918 20:39:50.044368 29472 ts_tablet_manager.cc:1085] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.000s I0918 20:39:50.046630 29472 raft_consensus.cc:340] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:50.054471 29472 raft_consensus.cc:366] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:50.054749 29472 raft_consensus.cc:705] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7c31a99ed64b749aa65aef3a705922, State: Initialized, Role: FOLLOWER I0918 20:39:50.056643 29470 ts_tablet_manager.cc:1113] T 413283585b6d41d5b6c0a0d032a86c37 P a0ce05b791dd4f2ba64117afb154edd9: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s I0918 20:39:50.055552 29472 consensus_queue.cc:229] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:50.059305 29472 ts_tablet_manager.cc:1113] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s I0918 20:39:50.075644 29510 tablet_bootstrap.cc:439] T 413283585b6d41d5b6c0a0d032a86c37 P e83f0943ebc44081a9df5fc862483865: No bootstrap required, opened a new log I0918 20:39:50.076400 29510 ts_tablet_manager.cc:1085] T 413283585b6d41d5b6c0a0d032a86c37 P e83f0943ebc44081a9df5fc862483865: Time spent bootstrapping tablet: real 0.042s user 0.011s sys 0.006s I0918 20:39:50.078903 29510 raft_consensus.cc:340] T 413283585b6d41d5b6c0a0d032a86c37 P e83f0943ebc44081a9df5fc862483865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:50.080015 29510 raft_consensus.cc:366] T 413283585b6d41d5b6c0a0d032a86c37 P e83f0943ebc44081a9df5fc862483865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:50.080284 29510 raft_consensus.cc:705] T 413283585b6d41d5b6c0a0d032a86c37 P e83f0943ebc44081a9df5fc862483865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e83f0943ebc44081a9df5fc862483865, State: Initialized, Role: FOLLOWER I0918 20:39:50.081039 29510 consensus_queue.cc:229] T 413283585b6d41d5b6c0a0d032a86c37 P e83f0943ebc44081a9df5fc862483865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:50.094197 29477 raft_consensus.cc:472] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:50.096647 29477 raft_consensus.cc:494] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:50.102416 29272 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "413283585b6d41d5b6c0a0d032a86c37" candidate_uuid: "bf7c31a99ed64b749aa65aef3a705922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0ce05b791dd4f2ba64117afb154edd9" is_pre_election: true I0918 20:39:50.103026 29479 consensus_queue.cc:984] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:39:50.103111 29272 raft_consensus.cc:2350] T 413283585b6d41d5b6c0a0d032a86c37 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf7c31a99ed64b749aa65aef3a705922 in term 0. I0918 20:39:50.106498 29477 leader_election.cc:296] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0ce05b791dd4f2ba64117afb154edd9 (127.18.222.1:38113), e83f0943ebc44081a9df5fc862483865 (127.18.222.3:37065) I0918 20:39:50.106768 29304 leader_election.cc:310] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0ce05b791dd4f2ba64117afb154edd9, bf7c31a99ed64b749aa65aef3a705922; no voters: I0918 20:39:50.107749 29477 raft_consensus.cc:2644] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:50.108064 29477 raft_consensus.cc:472] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:50.108379 29477 raft_consensus.cc:2890] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:50.108127 29426 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "413283585b6d41d5b6c0a0d032a86c37" candidate_uuid: "bf7c31a99ed64b749aa65aef3a705922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e83f0943ebc44081a9df5fc862483865" is_pre_election: true I0918 20:39:50.113131 29426 raft_consensus.cc:2350] T 413283585b6d41d5b6c0a0d032a86c37 P e83f0943ebc44081a9df5fc862483865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf7c31a99ed64b749aa65aef3a705922 in term 0. I0918 20:39:50.112174 29486 consensus_queue.cc:984] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:50.134707 29477 raft_consensus.cc:494] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:50.139745 29510 ts_tablet_manager.cc:1113] T 413283585b6d41d5b6c0a0d032a86c37 P e83f0943ebc44081a9df5fc862483865: Time spent starting tablet: real 0.063s user 0.011s sys 0.009s I0918 20:39:50.144259 29271 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "413283585b6d41d5b6c0a0d032a86c37" candidate_uuid: "bf7c31a99ed64b749aa65aef3a705922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0ce05b791dd4f2ba64117afb154edd9" I0918 20:39:50.145007 29271 raft_consensus.cc:2890] T 413283585b6d41d5b6c0a0d032a86c37 P a0ce05b791dd4f2ba64117afb154edd9 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:50.146270 29486 consensus_queue.cc:984] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:50.151571 29271 raft_consensus.cc:2350] T 413283585b6d41d5b6c0a0d032a86c37 P a0ce05b791dd4f2ba64117afb154edd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf7c31a99ed64b749aa65aef3a705922 in term 1. I0918 20:39:50.151686 29479 consensus_queue.cc:984] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0918 20:39:50.152830 29304 leader_election.cc:310] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0ce05b791dd4f2ba64117afb154edd9, bf7c31a99ed64b749aa65aef3a705922; no voters: I0918 20:39:50.153820 29486 raft_consensus.cc:2644] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:50.180053 29425 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "413283585b6d41d5b6c0a0d032a86c37" candidate_uuid: "bf7c31a99ed64b749aa65aef3a705922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e83f0943ebc44081a9df5fc862483865" I0918 20:39:50.181059 29425 raft_consensus.cc:2890] T 413283585b6d41d5b6c0a0d032a86c37 P e83f0943ebc44081a9df5fc862483865 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:50.191318 29425 raft_consensus.cc:2350] T 413283585b6d41d5b6c0a0d032a86c37 P e83f0943ebc44081a9df5fc862483865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf7c31a99ed64b749aa65aef3a705922 in term 1. I0918 20:39:50.199204 29477 leader_election.cc:296] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922 [CANDIDATE]: Term 1 election: Requested vote from peers a0ce05b791dd4f2ba64117afb154edd9 (127.18.222.1:38113), e83f0943ebc44081a9df5fc862483865 (127.18.222.3:37065) I0918 20:39:50.200445 29486 raft_consensus.cc:667] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922 [term 1 LEADER]: Becoming Leader. State: Replica: bf7c31a99ed64b749aa65aef3a705922, State: Running, Role: LEADER I0918 20:39:50.218291 29486 consensus_queue.cc:206] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } } I0918 20:39:50.283759 29519 consensus_queue.cc:984] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:50.286273 29519 consensus_queue.cc:984] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:50.336912 29184 catalog_manager.cc:4131] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922 reported cstate change: term changed from 0 to 1, leader changed from to bf7c31a99ed64b749aa65aef3a705922 (127.18.222.2). New cstate: current_term: 1 leader_uuid: "bf7c31a99ed64b749aa65aef3a705922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0ce05b791dd4f2ba64117afb154edd9" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf7c31a99ed64b749aa65aef3a705922" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 43609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e83f0943ebc44081a9df5fc862483865" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 37065 } health_report { overall_health: UNKNOWN } } } I0918 20:39:50.423787 29185 catalog_manager.cc:2376] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54972: table { table_name: "other-table" } alter_schema_steps { type: DROP_RANGE_PARTITION drop_range_partition { range_bounds { rows: "\006\001\000\000\000\000\007\001d\000\000\000" } } } schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "value" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } modify_external_catalogs: true I0918 20:39:50.454571 29185 catalog_manager.cc:4381] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac: Sending DeleteTablet for 3 replicas of tablet 413283585b6d41d5b6c0a0d032a86c37 I0918 20:39:50.455086 29185 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 413283585b6d41d5b6c0a0d032a86c37 on a0ce05b791dd4f2ba64117afb154edd9 (127.18.222.1:38113) (Partition dropped at 2019-09-18 20:39:50 UTC) I0918 20:39:50.455946 29185 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 413283585b6d41d5b6c0a0d032a86c37 on bf7c31a99ed64b749aa65aef3a705922 (127.18.222.2:43609) (Partition dropped at 2019-09-18 20:39:50 UTC) I0918 20:39:50.456602 29262 tablet_service.cc:1066] Processing DeleteTablet for tablet 413283585b6d41d5b6c0a0d032a86c37 with delete_type TABLET_DATA_DELETED (Partition dropped at 2019-09-18 20:39:50 UTC) from {username='slave'} at 127.0.0.1:53906 I0918 20:39:50.457299 29337 tablet_service.cc:1066] Processing DeleteTablet for tablet 413283585b6d41d5b6c0a0d032a86c37 with delete_type TABLET_DATA_DELETED (Partition dropped at 2019-09-18 20:39:50 UTC) from {username='slave'} at 127.0.0.1:49744 I0918 20:39:50.458091 29185 catalog_manager.cc:3523] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 413283585b6d41d5b6c0a0d032a86c37 on e83f0943ebc44081a9df5fc862483865 (127.18.222.3:37065) (Partition dropped at 2019-09-18 20:39:50 UTC) I0918 20:39:50.459683 29532 tablet_replica.cc:273] T 413283585b6d41d5b6c0a0d032a86c37 P a0ce05b791dd4f2ba64117afb154edd9: stopping tablet replica I0918 20:39:50.461077 29416 tablet_service.cc:1066] Processing DeleteTablet for tablet 413283585b6d41d5b6c0a0d032a86c37 with delete_type TABLET_DATA_DELETED (Partition dropped at 2019-09-18 20:39:50 UTC) from {username='slave'} at 127.0.0.1:43506 I0918 20:39:50.461164 29532 raft_consensus.cc:2147] T 413283585b6d41d5b6c0a0d032a86c37 P a0ce05b791dd4f2ba64117afb154edd9 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:50.461683 29532 raft_consensus.cc:2174] T 413283585b6d41d5b6c0a0d032a86c37 P a0ce05b791dd4f2ba64117afb154edd9 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:50.480453 29533 tablet_replica.cc:273] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922: stopping tablet replica I0918 20:39:50.482362 29533 raft_consensus.cc:2147] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:50.483114 29533 pending_rounds.cc:64] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922: Trying to abort 1 pending transactions. I0918 20:39:50.483330 29533 pending_rounds.cc:70] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6425965323376398336 op_type: NO_OP noop_request { } I0918 20:39:50.494000 29534 tablet_replica.cc:273] T 413283585b6d41d5b6c0a0d032a86c37 P e83f0943ebc44081a9df5fc862483865: stopping tablet replica I0918 20:39:50.494632 29534 raft_consensus.cc:2147] T 413283585b6d41d5b6c0a0d032a86c37 P e83f0943ebc44081a9df5fc862483865 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:50.494995 29534 raft_consensus.cc:2174] T 413283585b6d41d5b6c0a0d032a86c37 P e83f0943ebc44081a9df5fc862483865 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:50.496040 29533 raft_consensus.cc:2727] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted I0918 20:39:50.494468 29532 ts_tablet_manager.cc:1440] T 413283585b6d41d5b6c0a0d032a86c37 P a0ce05b791dd4f2ba64117afb154edd9: Deleting tablet data with delete state TABLET_DATA_DELETED I0918 20:39:50.502250 29534 ts_tablet_manager.cc:1440] T 413283585b6d41d5b6c0a0d032a86c37 P e83f0943ebc44081a9df5fc862483865: Deleting tablet data with delete state TABLET_DATA_DELETED I0918 20:39:50.496578 29533 raft_consensus.cc:2174] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:50.571228 29533 ts_tablet_manager.cc:1440] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922: Deleting tablet data with delete state TABLET_DATA_DELETED I0918 20:39:50.590936 29533 ts_tablet_manager.cc:1453] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 I0918 20:39:50.591761 29533 log.cc:990] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922: Deleting WAL directory at /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/ts-1-root/wals/413283585b6d41d5b6c0a0d032a86c37 I0918 20:39:50.593060 29533 ts_tablet_manager.cc:1474] T 413283585b6d41d5b6c0a0d032a86c37 P bf7c31a99ed64b749aa65aef3a705922: Deleting consensus metadata I0918 20:39:50.596433 29169 catalog_manager.cc:3501] TS bf7c31a99ed64b749aa65aef3a705922 (127.18.222.2:43609): tablet 413283585b6d41d5b6c0a0d032a86c37 (table other-table [id=a65a618babc844b18b588a6fde653e02]) successfully deleted I0918 20:39:50.608964 29532 ts_tablet_manager.cc:1453] T 413283585b6d41d5b6c0a0d032a86c37 P a0ce05b791dd4f2ba64117afb154edd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0 I0918 20:39:50.609549 29532 log.cc:990] T 413283585b6d41d5b6c0a0d032a86c37 P a0ce05b791dd4f2ba64117afb154edd9: Deleting WAL directory at /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/ts-0-root/wals/413283585b6d41d5b6c0a0d032a86c37 I0918 20:39:50.610316 29532 ts_tablet_manager.cc:1474] T 413283585b6d41d5b6c0a0d032a86c37 P a0ce05b791dd4f2ba64117afb154edd9: Deleting consensus metadata I0918 20:39:50.617481 29166 catalog_manager.cc:3501] TS a0ce05b791dd4f2ba64117afb154edd9 (127.18.222.1:38113): tablet 413283585b6d41d5b6c0a0d032a86c37 (table other-table [id=a65a618babc844b18b588a6fde653e02]) successfully deleted I0918 20:39:50.622390 29534 ts_tablet_manager.cc:1453] T 413283585b6d41d5b6c0a0d032a86c37 P e83f0943ebc44081a9df5fc862483865: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0 I0918 20:39:50.622987 29534 log.cc:990] T 413283585b6d41d5b6c0a0d032a86c37 P e83f0943ebc44081a9df5fc862483865: Deleting WAL directory at /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1568839089958813-19320/minicluster-data/ts-2-root/wals/413283585b6d41d5b6c0a0d032a86c37 I0918 20:39:50.623869 29534 ts_tablet_manager.cc:1474] T 413283585b6d41d5b6c0a0d032a86c37 P e83f0943ebc44081a9df5fc862483865: Deleting consensus metadata I0918 20:39:50.630038 29167 catalog_manager.cc:3501] TS e83f0943ebc44081a9df5fc862483865 (127.18.222.3:37065): tablet 413283585b6d41d5b6c0a0d032a86c37 (table other-table [id=a65a618babc844b18b588a6fde653e02]) successfully deleted I0918 20:39:51.612193 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:51.725940 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:51.726477 19320 tablet_replica.cc:273] T 7d73db44d1c64fab8cba98a77804d6c3 P a0ce05b791dd4f2ba64117afb154edd9: stopping tablet replica I0918 20:39:51.727010 19320 raft_consensus.cc:2147] T 7d73db44d1c64fab8cba98a77804d6c3 P a0ce05b791dd4f2ba64117afb154edd9 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:51.727512 19320 raft_consensus.cc:2174] T 7d73db44d1c64fab8cba98a77804d6c3 P a0ce05b791dd4f2ba64117afb154edd9 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:51.731878 19320 tablet_replica.cc:273] T d3484a1195b54f0c87d20ed7b6c38c32 P a0ce05b791dd4f2ba64117afb154edd9: stopping tablet replica I0918 20:39:51.732517 19320 raft_consensus.cc:2147] T d3484a1195b54f0c87d20ed7b6c38c32 P a0ce05b791dd4f2ba64117afb154edd9 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:51.732935 19320 raft_consensus.cc:2174] T d3484a1195b54f0c87d20ed7b6c38c32 P a0ce05b791dd4f2ba64117afb154edd9 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:51.736604 19320 tablet_replica.cc:273] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9: stopping tablet replica I0918 20:39:51.737176 19320 raft_consensus.cc:2147] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:51.738242 19320 raft_consensus.cc:2174] T c60b25182baf4c98b26d70efebd27734 P a0ce05b791dd4f2ba64117afb154edd9 [term 1 FOLLOWER]: Raft consensus is shut down! W0918 20:39:51.767590 29304 consensus_peers.cc:458] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 -> Peer a0ce05b791dd4f2ba64117afb154edd9 (127.18.222.1:38113): Couldn't send request to peer a0ce05b791dd4f2ba64117afb154edd9. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.1:38113: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:39:51.769309 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:51.769609 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:51.770052 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:51.796478 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:51.883437 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:51.883895 19320 tablet_replica.cc:273] T c60b25182baf4c98b26d70efebd27734 P bf7c31a99ed64b749aa65aef3a705922: stopping tablet replica I0918 20:39:51.884369 19320 raft_consensus.cc:2147] T c60b25182baf4c98b26d70efebd27734 P bf7c31a99ed64b749aa65aef3a705922 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:51.884753 19320 raft_consensus.cc:2174] T c60b25182baf4c98b26d70efebd27734 P bf7c31a99ed64b749aa65aef3a705922 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:51.888449 19320 tablet_replica.cc:273] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922: stopping tablet replica I0918 20:39:51.889031 19320 raft_consensus.cc:2147] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:51.889911 19320 raft_consensus.cc:2174] T d3484a1195b54f0c87d20ed7b6c38c32 P bf7c31a99ed64b749aa65aef3a705922 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:51.892557 19320 tablet_replica.cc:273] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922: stopping tablet replica I0918 20:39:51.892930 19320 raft_consensus.cc:2147] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:51.893689 19320 raft_consensus.cc:2174] T 7d73db44d1c64fab8cba98a77804d6c3 P bf7c31a99ed64b749aa65aef3a705922 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:51.923452 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:51.923662 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:51.924088 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:51.953155 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:52.002853 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:52.003386 19320 tablet_replica.cc:273] T 7d73db44d1c64fab8cba98a77804d6c3 P e83f0943ebc44081a9df5fc862483865: stopping tablet replica I0918 20:39:52.004106 19320 raft_consensus.cc:2147] T 7d73db44d1c64fab8cba98a77804d6c3 P e83f0943ebc44081a9df5fc862483865 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:52.004570 19320 raft_consensus.cc:2174] T 7d73db44d1c64fab8cba98a77804d6c3 P e83f0943ebc44081a9df5fc862483865 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:52.008098 19320 tablet_replica.cc:273] T d3484a1195b54f0c87d20ed7b6c38c32 P e83f0943ebc44081a9df5fc862483865: stopping tablet replica I0918 20:39:52.008661 19320 raft_consensus.cc:2147] T d3484a1195b54f0c87d20ed7b6c38c32 P e83f0943ebc44081a9df5fc862483865 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:52.009094 19320 raft_consensus.cc:2174] T d3484a1195b54f0c87d20ed7b6c38c32 P e83f0943ebc44081a9df5fc862483865 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:52.012367 19320 tablet_replica.cc:273] T c60b25182baf4c98b26d70efebd27734 P e83f0943ebc44081a9df5fc862483865: stopping tablet replica I0918 20:39:52.012915 19320 raft_consensus.cc:2147] T c60b25182baf4c98b26d70efebd27734 P e83f0943ebc44081a9df5fc862483865 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:52.013303 19320 raft_consensus.cc:2174] T c60b25182baf4c98b26d70efebd27734 P e83f0943ebc44081a9df5fc862483865 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:52.040254 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:52.040489 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:52.040894 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:52.061792 19320 master.cc:247] Master@127.18.222.62:40629 shutting down... I0918 20:39:52.093688 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:52.094233 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:52.094617 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 38753c8ea6244f958281a39c906cd411: stopping tablet replica I0918 20:39:52.117386 19320 master.cc:258] Master@127.18.222.62:40629 shutdown complete. I0918 20:39:52.130350 19320 master.cc:247] Master@127.18.222.61:39277 shutting down... I0918 20:39:52.163887 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:52.164443 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:52.164829 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 6e0a29e7c61e40d0a4293b62ec0ac59a: stopping tablet replica I0918 20:39:52.186602 19320 master.cc:258] Master@127.18.222.61:39277 shutdown complete. W0918 20:39:52.195339 29167 consensus_peers.cc:458] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac -> Peer 38753c8ea6244f958281a39c906cd411 (127.18.222.62:40629): Couldn't send request to peer 38753c8ea6244f958281a39c906cd411. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.62:40629: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:39:52.202706 19320 master.cc:247] Master@127.18.222.60:39727 shutting down... I0918 20:39:52.233165 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:52.233983 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:52.234270 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 50f4f536877e4dc3b4f97c7d7a214cac: stopping tablet replica W0918 20:39:53.242837 19320 thread.cc:499] Waited for 1000ms trying to join with diag-logger (tid 29170) I0918 20:39:53.598033 19320 master.cc:258] Master@127.18.222.60:39727 shutdown complete. [ OK ] RemoteKsckTest.TestTableFilters (5941 ms) [ RUN ] RemoteKsckTest.TestTabletFiltersNoMatch I0918 20:39:53.651422 19320 test_util.cc:212] Using random seed: -690427714 I0918 20:39:53.652343 19320 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.18.222.62:45907,127.18.222.61:44027,127.18.222.60:41777 I0918 20:39:53.665661 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 174092us I0918 20:39:53.666041 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:53.666224 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:53.666453 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:53.666581 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:53.667788 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/master-0-root/instance: uuid: "f42f36621b204e808330eb34ce6c6842" format_stamp: "Formatted at 2019-09-18 20:39:53 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:53.676604 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s I0918 20:39:53.678153 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:53.684986 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0918 20:39:53.685222 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/master-0-root uuid: "f42f36621b204e808330eb34ce6c6842" format_stamp: "Formatted at 2019-09-18 20:39:53 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:53.685479 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:53.742969 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:45907 I0918 20:39:53.745107 19320 webserver.cc:374] Webserver started at http://127.18.222.62:38997/ using document root and password file I0918 20:39:53.747550 29631 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:53.757752 29631 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:53.769177 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 174092us I0918 20:39:53.769567 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:53.769752 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:53.769973 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:53.770093 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:53.771461 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/master-1-root/instance: uuid: "9524c898626646999a750e57a583d010" format_stamp: "Formatted at 2019-09-18 20:39:53 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:53.776072 29631 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:53.777992 29631 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:44027: Network error: Client connection negotiation failed: client connection to 127.18.222.61:44027: connect: Connection refused (error 111) I0918 20:39:53.783013 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.010s sys 0.001s I0918 20:39:53.784812 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:53.792695 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s I0918 20:39:53.792878 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/master-1-root uuid: "9524c898626646999a750e57a583d010" format_stamp: "Formatted at 2019-09-18 20:39:53 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:53.793109 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:53.802417 29631 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } attempt: 1 W0918 20:39:53.804497 29631 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:44027: Network error: Client connection negotiation failed: client connection to 127.18.222.61:44027: connect: Connection refused (error 111) I0918 20:39:53.874969 29631 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } attempt: 2 W0918 20:39:53.877344 29631 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:44027: Network error: Client connection negotiation failed: client connection to 127.18.222.61:44027: connect: Connection refused (error 111) I0918 20:39:53.885996 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:44027 I0918 20:39:53.888090 19320 webserver.cc:374] Webserver started at http://127.18.222.61:33367/ using document root and password file I0918 20:39:53.889876 29688 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:53.896014 29688 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:53.909440 29688 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:53.916669 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 174092us I0918 20:39:53.917062 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:53.917232 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:53.917452 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:53.917639 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:53.919044 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/master-2-root/instance: uuid: "510fdaab3c2f457bb828ac464cf414ae" format_stamp: "Formatted at 2019-09-18 20:39:53 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:53.923563 29688 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:53.926228 29688 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:41777: Network error: Client connection negotiation failed: client connection to 127.18.222.60:41777: connect: Connection refused (error 111) I0918 20:39:53.932310 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.013s user 0.009s sys 0.006s I0918 20:39:53.934022 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:53.942003 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:53.942211 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/master-2-root uuid: "510fdaab3c2f457bb828ac464cf414ae" format_stamp: "Formatted at 2019-09-18 20:39:53 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:53.942450 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:53.951670 29688 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } attempt: 1 W0918 20:39:53.953713 29688 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:41777: Network error: Client connection negotiation failed: client connection to 127.18.222.60:41777: connect: Connection refused (error 111) I0918 20:39:53.957754 29631 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } attempt: 3 I0918 20:39:53.970521 29631 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:53.972559 29631 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:41777: Network error: Client connection negotiation failed: client connection to 127.18.222.60:41777: connect: Connection refused (error 111) I0918 20:39:54.011030 29631 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } attempt: 1 W0918 20:39:54.013819 29631 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:41777: Network error: Client connection negotiation failed: client connection to 127.18.222.60:41777: connect: Connection refused (error 111) I0918 20:39:54.022089 29688 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } attempt: 2 W0918 20:39:54.024535 29688 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:41777: Network error: Client connection negotiation failed: client connection to 127.18.222.60:41777: connect: Connection refused (error 111) I0918 20:39:54.037890 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:41777 I0918 20:39:54.039911 19320 webserver.cc:374] Webserver started at http://127.18.222.60:39721/ using document root and password file I0918 20:39:54.041702 29745 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:54.043792 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0918 20:39:54.048041 29745 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:54.059350 29745 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:54.067252 29631 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } attempt: 2 I0918 20:39:54.071238 29745 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:54.087759 29631 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842: Bootstrap starting. I0918 20:39:54.092389 29631 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842: No blocks or log segments found. Creating new log. I0918 20:39:54.095286 29745 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae: Bootstrap starting. I0918 20:39:54.096757 29631 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842: No bootstrap required, opened a new log I0918 20:39:54.099668 29745 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae: No blocks or log segments found. Creating new log. I0918 20:39:54.099529 29631 raft_consensus.cc:340] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } I0918 20:39:54.100428 29631 raft_consensus.cc:366] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:54.100685 29631 raft_consensus.cc:705] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42f36621b204e808330eb34ce6c6842, State: Initialized, Role: FOLLOWER I0918 20:39:54.100975 29688 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } attempt: 3 I0918 20:39:54.101635 29631 consensus_queue.cc:229] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } I0918 20:39:54.104485 29745 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae: No bootstrap required, opened a new log I0918 20:39:54.105100 29751 sys_catalog.cc:340] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } } I0918 20:39:54.105926 29751 sys_catalog.cc:343] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:54.106523 29631 sys_catalog.cc:340] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } } I0918 20:39:54.107192 29631 sys_catalog.cc:343] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:54.107276 29745 raft_consensus.cc:340] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } I0918 20:39:54.108194 29745 raft_consensus.cc:366] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:54.108419 29745 raft_consensus.cc:705] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 510fdaab3c2f457bb828ac464cf414ae, State: Initialized, Role: FOLLOWER I0918 20:39:54.108543 29631 sys_catalog.cc:428] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:54.109949 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 I0918 20:39:54.109283 29745 consensus_queue.cc:229] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } W0918 20:39:54.113404 29752 catalog_manager.cc:1155] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:54.113797 29752 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:54.113307 29745 sys_catalog.cc:340] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } } I0918 20:39:54.114089 29745 sys_catalog.cc:343] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:54.113569 29753 sys_catalog.cc:340] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } } I0918 20:39:54.114398 29753 sys_catalog.cc:343] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:54.115154 29745 sys_catalog.cc:428] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [sys.catalog]: configured and running, proceeding with master startup. W0918 20:39:54.118316 29754 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:54.118512 29754 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:54.118458 29688 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010: Bootstrap starting. I0918 20:39:54.121423 29688 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010: No blocks or log segments found. Creating new log. I0918 20:39:54.124425 29688 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010: No bootstrap required, opened a new log I0918 20:39:54.126013 29688 raft_consensus.cc:340] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } I0918 20:39:54.126493 29688 raft_consensus.cc:366] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:54.126672 29688 raft_consensus.cc:705] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9524c898626646999a750e57a583d010, State: Initialized, Role: FOLLOWER I0918 20:39:54.127270 29688 consensus_queue.cc:229] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } I0918 20:39:54.128906 29756 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } } I0918 20:39:54.129410 29756 sys_catalog.cc:343] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:54.129869 29688 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } } I0918 20:39:54.130324 29688 sys_catalog.cc:343] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:54.131228 29688 sys_catalog.cc:428] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:54.132269 29757 catalog_manager.cc:569] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0918 20:39:54.132623 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 I0918 20:39:54.148497 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 174592us I0918 20:39:54.148926 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:54.149083 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:54.149317 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:54.149441 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:54.150524 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-0-root/instance: uuid: "04da295022bf4d77beae5119e3db41f5" format_stamp: "Formatted at 2019-09-18 20:39:54 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:54.160249 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.006s sys 0.005s I0918 20:39:54.161876 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:54.169891 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:54.170109 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-0-root uuid: "04da295022bf4d77beae5119e3db41f5" format_stamp: "Formatted at 2019-09-18 20:39:54 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:54.170392 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:54.187482 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:54.187649 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:54.225225 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:41197 I0918 20:39:54.227317 19320 webserver.cc:374] Webserver started at http://127.18.222.1:41247/ using document root and password file I0918 20:39:54.254467 29822 heartbeater.cc:322] Connected to a master server at 127.18.222.60:41777 I0918 20:39:54.254930 29822 heartbeater.cc:406] Registering TS with master... I0918 20:39:54.254765 29823 heartbeater.cc:322] Connected to a master server at 127.18.222.61:44027 I0918 20:39:54.255518 29823 heartbeater.cc:406] Registering TS with master... I0918 20:39:54.255973 29822 heartbeater.cc:455] Master 127.18.222.60:41777 requested a full tablet report, sending... I0918 20:39:54.256081 29823 heartbeater.cc:455] Master 127.18.222.61:44027 requested a full tablet report, sending... I0918 20:39:54.259093 29825 heartbeater.cc:322] Connected to a master server at 127.18.222.62:45907 I0918 20:39:54.259413 29825 heartbeater.cc:406] Registering TS with master... I0918 20:39:54.259260 29655 ts_manager.cc:186] Registered new tserver with Master: 04da295022bf4d77beae5119e3db41f5 (127.18.222.1:41197) I0918 20:39:54.259271 29712 ts_manager.cc:186] Registered new tserver with Master: 04da295022bf4d77beae5119e3db41f5 (127.18.222.1:41197) I0918 20:39:54.260272 29825 heartbeater.cc:455] Master 127.18.222.62:45907 requested a full tablet report, sending... I0918 20:39:54.262477 29598 ts_manager.cc:186] Registered new tserver with Master: 04da295022bf4d77beae5119e3db41f5 (127.18.222.1:41197) I0918 20:39:54.267122 29753 raft_consensus.cc:472] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:54.267402 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 174592us I0918 20:39:54.267899 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:54.267670 29753 raft_consensus.cc:494] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } I0918 20:39:54.268157 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:54.268445 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:54.268580 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:54.269294 29753 leader_election.cc:296] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f42f36621b204e808330eb34ce6c6842 (127.18.222.62:45907), 9524c898626646999a750e57a583d010 (127.18.222.61:44027) I0918 20:39:54.269969 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-1-root/instance: uuid: "1c925d542dfc4c38b7f68380781223b1" format_stamp: "Formatted at 2019-09-18 20:39:54 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:54.270401 29665 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "510fdaab3c2f457bb828ac464cf414ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9524c898626646999a750e57a583d010" is_pre_election: true I0918 20:39:54.271474 29665 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 510fdaab3c2f457bb828ac464cf414ae in term 0. I0918 20:39:54.270309 29608 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "510fdaab3c2f457bb828ac464cf414ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42f36621b204e808330eb34ce6c6842" is_pre_election: true I0918 20:39:54.272483 29608 raft_consensus.cc:2350] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 510fdaab3c2f457bb828ac464cf414ae in term 0. I0918 20:39:54.272688 29696 leader_election.cc:310] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 510fdaab3c2f457bb828ac464cf414ae, 9524c898626646999a750e57a583d010; no voters: I0918 20:39:54.273821 29753 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:54.274128 29753 raft_consensus.cc:472] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:54.274410 29753 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:54.280534 29753 raft_consensus.cc:494] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } I0918 20:39:54.281754 29753 leader_election.cc:296] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [CANDIDATE]: Term 1 election: Requested vote from peers f42f36621b204e808330eb34ce6c6842 (127.18.222.62:45907), 9524c898626646999a750e57a583d010 (127.18.222.61:44027) I0918 20:39:54.282675 29608 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "510fdaab3c2f457bb828ac464cf414ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42f36621b204e808330eb34ce6c6842" I0918 20:39:54.282819 29665 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "510fdaab3c2f457bb828ac464cf414ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9524c898626646999a750e57a583d010" I0918 20:39:54.283233 29608 raft_consensus.cc:2890] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:54.283560 29665 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:54.284842 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.014s user 0.012s sys 0.002s I0918 20:39:54.287792 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:54.289382 29608 raft_consensus.cc:2350] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 510fdaab3c2f457bb828ac464cf414ae in term 1. I0918 20:39:54.290784 29693 leader_election.cc:310] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 510fdaab3c2f457bb828ac464cf414ae, f42f36621b204e808330eb34ce6c6842; no voters: I0918 20:39:54.291893 29753 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:54.296631 29665 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 510fdaab3c2f457bb828ac464cf414ae in term 1. I0918 20:39:54.299432 29753 raft_consensus.cc:667] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [term 1 LEADER]: Becoming Leader. State: Replica: 510fdaab3c2f457bb828ac464cf414ae, State: Running, Role: LEADER I0918 20:39:54.300207 29753 consensus_queue.cc:206] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } I0918 20:39:54.301973 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:54.302314 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-1-root uuid: "1c925d542dfc4c38b7f68380781223b1" format_stamp: "Formatted at 2019-09-18 20:39:54 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:54.302640 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:54.305891 29830 sys_catalog.cc:340] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 510fdaab3c2f457bb828ac464cf414ae. Latest consensus state: current_term: 1 leader_uuid: "510fdaab3c2f457bb828ac464cf414ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } } I0918 20:39:54.307196 29830 sys_catalog.cc:343] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [sys.catalog]: This master's current role is: LEADER I0918 20:39:54.310451 29834 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:39:54.314644 29834 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:39:54.349088 29665 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010 [term 1 FOLLOWER]: Refusing update from remote peer 510fdaab3c2f457bb828ac464cf414ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:54.350721 29830 consensus_queue.cc:984] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:54.350875 29608 raft_consensus.cc:1184] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842 [term 1 FOLLOWER]: Refusing update from remote peer 510fdaab3c2f457bb828ac464cf414ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:54.352383 29830 consensus_queue.cc:984] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:54.366626 29756 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 510fdaab3c2f457bb828ac464cf414ae. Latest consensus state: current_term: 1 leader_uuid: "510fdaab3c2f457bb828ac464cf414ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } } I0918 20:39:54.368119 29756 sys_catalog.cc:343] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:54.370635 29751 sys_catalog.cc:340] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 510fdaab3c2f457bb828ac464cf414ae. Latest consensus state: current_term: 1 leader_uuid: "510fdaab3c2f457bb828ac464cf414ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } } I0918 20:39:54.372668 29751 sys_catalog.cc:343] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:54.379189 29838 mvcc.cc:203] Tried to move safe_time back from 6425965340037177344 to 6425965339862011904. Current Snapshot: MvccSnapshot[committed={T|T < 6425965340037177344}] I0918 20:39:54.386001 29751 sys_catalog.cc:340] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "510fdaab3c2f457bb828ac464cf414ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } } I0918 20:39:54.387432 29751 sys_catalog.cc:343] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:54.390878 29830 sys_catalog.cc:340] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "510fdaab3c2f457bb828ac464cf414ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } } I0918 20:39:54.392704 29830 sys_catalog.cc:343] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [sys.catalog]: This master's current role is: LEADER I0918 20:39:54.393249 29830 sys_catalog.cc:340] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "510fdaab3c2f457bb828ac464cf414ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } } I0918 20:39:54.394438 29830 sys_catalog.cc:343] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [sys.catalog]: This master's current role is: LEADER I0918 20:39:54.397641 29834 catalog_manager.cc:981] Generated new certificate authority record I0918 20:39:54.400111 29756 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "510fdaab3c2f457bb828ac464cf414ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42f36621b204e808330eb34ce6c6842" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 45907 } } peers { permanent_uuid: "9524c898626646999a750e57a583d010" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 44027 } } peers { permanent_uuid: "510fdaab3c2f457bb828ac464cf414ae" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 41777 } } } I0918 20:39:54.401150 29756 sys_catalog.cc:343] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:54.403970 29834 catalog_manager.cc:1106] Loading token signing keys... I0918 20:39:54.405390 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:54.405648 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:54.431792 29834 catalog_manager.cc:4446] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae: Generated new TSK 0 I0918 20:39:54.432682 29834 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:39:54.444895 29712 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43482 I0918 20:39:54.458272 29822 heartbeater.cc:447] Master 127.18.222.60:41777 was elected leader, sending a full tablet report... I0918 20:39:54.483595 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.2:40767 I0918 20:39:54.486937 19320 webserver.cc:374] Webserver started at http://127.18.222.2:42963/ using document root and password file I0918 20:39:54.534564 29906 heartbeater.cc:322] Connected to a master server at 127.18.222.61:44027 I0918 20:39:54.535073 29906 heartbeater.cc:406] Registering TS with master... I0918 20:39:54.535871 29906 heartbeater.cc:455] Master 127.18.222.61:44027 requested a full tablet report, sending... I0918 20:39:54.538754 29655 ts_manager.cc:186] Registered new tserver with Master: 1c925d542dfc4c38b7f68380781223b1 (127.18.222.2:40767) I0918 20:39:54.541441 29905 heartbeater.cc:322] Connected to a master server at 127.18.222.60:41777 I0918 20:39:54.541537 29907 heartbeater.cc:322] Connected to a master server at 127.18.222.62:45907 I0918 20:39:54.541813 29905 heartbeater.cc:406] Registering TS with master... I0918 20:39:54.541999 29907 heartbeater.cc:406] Registering TS with master... I0918 20:39:54.542826 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 174592us I0918 20:39:54.542961 29905 heartbeater.cc:455] Master 127.18.222.60:41777 requested a full tablet report, sending... I0918 20:39:54.543028 29907 heartbeater.cc:455] Master 127.18.222.62:45907 requested a full tablet report, sending... I0918 20:39:54.543517 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:54.543870 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:54.545563 29712 ts_manager.cc:186] Registered new tserver with Master: 1c925d542dfc4c38b7f68380781223b1 (127.18.222.2:40767) I0918 20:39:54.546804 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:54.547040 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:54.546075 29598 ts_manager.cc:186] Registered new tserver with Master: 1c925d542dfc4c38b7f68380781223b1 (127.18.222.2:40767) I0918 20:39:54.549234 29712 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43488 I0918 20:39:54.551061 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-2-root/instance: uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" format_stamp: "Formatted at 2019-09-18 20:39:54 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:54.562817 29905 heartbeater.cc:447] Master 127.18.222.60:41777 was elected leader, sending a full tablet report... I0918 20:39:54.565826 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.014s user 0.011s sys 0.002s I0918 20:39:54.568069 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:54.579632 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0918 20:39:54.579973 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-2-root uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" format_stamp: "Formatted at 2019-09-18 20:39:54 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:54.580299 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1568839089958813-19320/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:54.663408 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:54.663650 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:54.726050 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.3:44951 I0918 20:39:54.727942 19320 webserver.cc:374] Webserver started at http://127.18.222.3:37847/ using document root and password file I0918 20:39:54.778148 29977 heartbeater.cc:322] Connected to a master server at 127.18.222.61:44027 I0918 20:39:54.778550 29977 heartbeater.cc:406] Registering TS with master... I0918 20:39:54.779917 29977 heartbeater.cc:455] Master 127.18.222.61:44027 requested a full tablet report, sending... I0918 20:39:54.782382 29654 ts_manager.cc:186] Registered new tserver with Master: 55a18fe3cda74154bd8e16d6ffc31cfd (127.18.222.3:44951) I0918 20:39:54.786213 29976 heartbeater.cc:322] Connected to a master server at 127.18.222.60:41777 I0918 20:39:54.786557 29976 heartbeater.cc:406] Registering TS with master... I0918 20:39:54.787125 29976 heartbeater.cc:455] Master 127.18.222.60:41777 requested a full tablet report, sending... I0918 20:39:54.788985 29980 heartbeater.cc:322] Connected to a master server at 127.18.222.62:45907 I0918 20:39:54.789031 29711 ts_manager.cc:186] Registered new tserver with Master: 55a18fe3cda74154bd8e16d6ffc31cfd (127.18.222.3:44951) I0918 20:39:54.789377 29980 heartbeater.cc:406] Registering TS with master... I0918 20:39:54.792348 29980 heartbeater.cc:455] Master 127.18.222.62:45907 requested a full tablet report, sending... I0918 20:39:54.793493 29711 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43494 I0918 20:39:54.795651 29598 ts_manager.cc:186] Registered new tserver with Master: 55a18fe3cda74154bd8e16d6ffc31cfd (127.18.222.3:44951) I0918 20:39:54.795787 19320 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.05255265s I0918 20:39:54.808547 29976 heartbeater.cc:447] Master 127.18.222.60:41777 was elected leader, sending a full tablet report... I0918 20:39:54.853152 29711 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43502: name: "ksck-test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\n\000\000\000\004\001\024\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:39:54.855494 29711 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:39:54.972693 29789 tablet_service.cc:1022] Processing CreateTablet for tablet 7da02c564f274cf3a56af0d48c23b019 (table=ksck-test-table [id=15bece6c97d34c24b90f775d29ab2456]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:54.974004 29789 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 7da02c564f274cf3a56af0d48c23b019. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:54.981838 29943 tablet_service.cc:1022] Processing CreateTablet for tablet 7da02c564f274cf3a56af0d48c23b019 (table=ksck-test-table [id=15bece6c97d34c24b90f775d29ab2456]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:54.983109 29943 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 7da02c564f274cf3a56af0d48c23b019. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:54.988903 29942 tablet_service.cc:1022] Processing CreateTablet for tablet cd97b4ac0e56420ebf3dc1398599f0b1 (table=ksck-test-table [id=15bece6c97d34c24b90f775d29ab2456]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:54.990187 29942 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet cd97b4ac0e56420ebf3dc1398599f0b1. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:54.990212 29872 tablet_service.cc:1022] Processing CreateTablet for tablet 7da02c564f274cf3a56af0d48c23b019 (table=ksck-test-table [id=15bece6c97d34c24b90f775d29ab2456]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:54.991226 29872 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 7da02c564f274cf3a56af0d48c23b019. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:54.994264 29941 tablet_service.cc:1022] Processing CreateTablet for tablet de78d398560141c6977d2344fc5c65db (table=ksck-test-table [id=15bece6c97d34c24b90f775d29ab2456]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:54.995513 29941 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet de78d398560141c6977d2344fc5c65db. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:54.996104 29871 tablet_service.cc:1022] Processing CreateTablet for tablet cd97b4ac0e56420ebf3dc1398599f0b1 (table=ksck-test-table [id=15bece6c97d34c24b90f775d29ab2456]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:54.997166 29871 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet cd97b4ac0e56420ebf3dc1398599f0b1. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:55.007076 29870 tablet_service.cc:1022] Processing CreateTablet for tablet de78d398560141c6977d2344fc5c65db (table=ksck-test-table [id=15bece6c97d34c24b90f775d29ab2456]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:55.008486 29870 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet de78d398560141c6977d2344fc5c65db. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:55.017140 29788 tablet_service.cc:1022] Processing CreateTablet for tablet cd97b4ac0e56420ebf3dc1398599f0b1 (table=ksck-test-table [id=15bece6c97d34c24b90f775d29ab2456]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:55.018417 29788 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet cd97b4ac0e56420ebf3dc1398599f0b1. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:55.025383 29787 tablet_service.cc:1022] Processing CreateTablet for tablet de78d398560141c6977d2344fc5c65db (table=ksck-test-table [id=15bece6c97d34c24b90f775d29ab2456]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:55.026592 29787 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet de78d398560141c6977d2344fc5c65db. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:55.085238 29997 tablet_bootstrap.cc:439] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd: Bootstrap starting. I0918 20:39:55.105803 29998 tablet_bootstrap.cc:439] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5: Bootstrap starting. I0918 20:39:55.120651 29752 catalog_manager.cc:1153] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842: acquiring CA information for follower catalog manager: success I0918 20:39:55.125396 29999 tablet_bootstrap.cc:439] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1: Bootstrap starting. I0918 20:39:55.127297 29997 tablet_bootstrap.cc:590] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd: No blocks or log segments found. Creating new log. I0918 20:39:55.140825 29757 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010: acquiring CA information for follower catalog manager: success I0918 20:39:55.133167 29752 catalog_manager.cc:1181] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:55.143489 29998 tablet_bootstrap.cc:590] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5: No blocks or log segments found. Creating new log. I0918 20:39:55.144899 29999 tablet_bootstrap.cc:590] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1: No blocks or log segments found. Creating new log. I0918 20:39:55.151007 29757 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:55.165077 29997 tablet_bootstrap.cc:439] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd: No bootstrap required, opened a new log I0918 20:39:55.165642 29997 ts_tablet_manager.cc:1085] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd: Time spent bootstrapping tablet: real 0.081s user 0.000s sys 0.010s I0918 20:39:55.170428 29998 tablet_bootstrap.cc:439] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5: No bootstrap required, opened a new log I0918 20:39:55.168287 29997 raft_consensus.cc:340] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } I0918 20:39:55.171154 29998 ts_tablet_manager.cc:1085] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.008s I0918 20:39:55.171213 29997 raft_consensus.cc:366] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:55.173885 29998 raft_consensus.cc:340] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } I0918 20:39:55.174787 29998 raft_consensus.cc:366] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:55.175070 29998 raft_consensus.cc:705] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04da295022bf4d77beae5119e3db41f5, State: Initialized, Role: FOLLOWER I0918 20:39:55.175927 29998 consensus_queue.cc:229] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } I0918 20:39:55.170344 29999 tablet_bootstrap.cc:439] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1: No bootstrap required, opened a new log I0918 20:39:55.171651 29997 raft_consensus.cc:705] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a18fe3cda74154bd8e16d6ffc31cfd, State: Initialized, Role: FOLLOWER I0918 20:39:55.184590 29999 ts_tablet_manager.cc:1085] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1: Time spent bootstrapping tablet: real 0.059s user 0.000s sys 0.023s I0918 20:39:55.187291 29999 raft_consensus.cc:340] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } I0918 20:39:55.187844 29999 raft_consensus.cc:366] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:55.187741 29998 ts_tablet_manager.cc:1113] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5: Time spent starting tablet: real 0.016s user 0.006s sys 0.006s I0918 20:39:55.188053 29999 raft_consensus.cc:705] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c925d542dfc4c38b7f68380781223b1, State: Initialized, Role: FOLLOWER I0918 20:39:55.188787 29998 tablet_bootstrap.cc:439] T cd97b4ac0e56420ebf3dc1398599f0b1 P 04da295022bf4d77beae5119e3db41f5: Bootstrap starting. I0918 20:39:55.188792 29999 consensus_queue.cc:229] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } I0918 20:39:55.193478 29998 tablet_bootstrap.cc:590] T cd97b4ac0e56420ebf3dc1398599f0b1 P 04da295022bf4d77beae5119e3db41f5: No blocks or log segments found. Creating new log. I0918 20:39:55.194937 29999 ts_tablet_manager.cc:1113] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1: Time spent starting tablet: real 0.010s user 0.004s sys 0.003s I0918 20:39:55.195924 29999 tablet_bootstrap.cc:439] T 7da02c564f274cf3a56af0d48c23b019 P 1c925d542dfc4c38b7f68380781223b1: Bootstrap starting. I0918 20:39:55.202792 29998 tablet_bootstrap.cc:439] T cd97b4ac0e56420ebf3dc1398599f0b1 P 04da295022bf4d77beae5119e3db41f5: No bootstrap required, opened a new log I0918 20:39:55.203547 29998 ts_tablet_manager.cc:1085] T cd97b4ac0e56420ebf3dc1398599f0b1 P 04da295022bf4d77beae5119e3db41f5: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s I0918 20:39:55.204790 29999 tablet_bootstrap.cc:590] T 7da02c564f274cf3a56af0d48c23b019 P 1c925d542dfc4c38b7f68380781223b1: No blocks or log segments found. Creating new log. I0918 20:39:55.206064 29998 raft_consensus.cc:340] T cd97b4ac0e56420ebf3dc1398599f0b1 P 04da295022bf4d77beae5119e3db41f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } I0918 20:39:55.206923 29998 raft_consensus.cc:366] T cd97b4ac0e56420ebf3dc1398599f0b1 P 04da295022bf4d77beae5119e3db41f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:55.207188 29998 raft_consensus.cc:705] T cd97b4ac0e56420ebf3dc1398599f0b1 P 04da295022bf4d77beae5119e3db41f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04da295022bf4d77beae5119e3db41f5, State: Initialized, Role: FOLLOWER I0918 20:39:55.208513 29998 consensus_queue.cc:229] T cd97b4ac0e56420ebf3dc1398599f0b1 P 04da295022bf4d77beae5119e3db41f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } I0918 20:39:55.210408 29998 ts_tablet_manager.cc:1113] T cd97b4ac0e56420ebf3dc1398599f0b1 P 04da295022bf4d77beae5119e3db41f5: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s I0918 20:39:55.211131 29998 tablet_bootstrap.cc:439] T de78d398560141c6977d2344fc5c65db P 04da295022bf4d77beae5119e3db41f5: Bootstrap starting. I0918 20:39:55.214922 29998 tablet_bootstrap.cc:590] T de78d398560141c6977d2344fc5c65db P 04da295022bf4d77beae5119e3db41f5: No blocks or log segments found. Creating new log. I0918 20:39:55.217414 29997 consensus_queue.cc:229] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } I0918 20:39:55.227761 29998 tablet_bootstrap.cc:439] T de78d398560141c6977d2344fc5c65db P 04da295022bf4d77beae5119e3db41f5: No bootstrap required, opened a new log I0918 20:39:55.228433 29998 ts_tablet_manager.cc:1085] T de78d398560141c6977d2344fc5c65db P 04da295022bf4d77beae5119e3db41f5: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.007s I0918 20:39:55.230717 29999 tablet_bootstrap.cc:439] T 7da02c564f274cf3a56af0d48c23b019 P 1c925d542dfc4c38b7f68380781223b1: No bootstrap required, opened a new log I0918 20:39:55.231472 29999 ts_tablet_manager.cc:1085] T 7da02c564f274cf3a56af0d48c23b019 P 1c925d542dfc4c38b7f68380781223b1: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.004s I0918 20:39:55.242238 29998 raft_consensus.cc:340] T de78d398560141c6977d2344fc5c65db P 04da295022bf4d77beae5119e3db41f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } I0918 20:39:55.244107 29998 raft_consensus.cc:366] T de78d398560141c6977d2344fc5c65db P 04da295022bf4d77beae5119e3db41f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:55.244316 29998 raft_consensus.cc:705] T de78d398560141c6977d2344fc5c65db P 04da295022bf4d77beae5119e3db41f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04da295022bf4d77beae5119e3db41f5, State: Initialized, Role: FOLLOWER I0918 20:39:55.244963 29998 consensus_queue.cc:229] T de78d398560141c6977d2344fc5c65db P 04da295022bf4d77beae5119e3db41f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } I0918 20:39:55.246415 29998 ts_tablet_manager.cc:1113] T de78d398560141c6977d2344fc5c65db P 04da295022bf4d77beae5119e3db41f5: Time spent starting tablet: real 0.018s user 0.003s sys 0.001s I0918 20:39:55.250738 29997 ts_tablet_manager.cc:1113] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd: Time spent starting tablet: real 0.085s user 0.006s sys 0.003s I0918 20:39:55.255506 29999 raft_consensus.cc:340] T 7da02c564f274cf3a56af0d48c23b019 P 1c925d542dfc4c38b7f68380781223b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } I0918 20:39:55.256448 29999 raft_consensus.cc:366] T 7da02c564f274cf3a56af0d48c23b019 P 1c925d542dfc4c38b7f68380781223b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:55.256729 29999 raft_consensus.cc:705] T 7da02c564f274cf3a56af0d48c23b019 P 1c925d542dfc4c38b7f68380781223b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c925d542dfc4c38b7f68380781223b1, State: Initialized, Role: FOLLOWER I0918 20:39:55.257588 29999 consensus_queue.cc:229] T 7da02c564f274cf3a56af0d48c23b019 P 1c925d542dfc4c38b7f68380781223b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } I0918 20:39:55.258179 29997 tablet_bootstrap.cc:439] T 7da02c564f274cf3a56af0d48c23b019 P 55a18fe3cda74154bd8e16d6ffc31cfd: Bootstrap starting. I0918 20:39:55.262986 29997 tablet_bootstrap.cc:590] T 7da02c564f274cf3a56af0d48c23b019 P 55a18fe3cda74154bd8e16d6ffc31cfd: No blocks or log segments found. Creating new log. I0918 20:39:55.272027 29997 tablet_bootstrap.cc:439] T 7da02c564f274cf3a56af0d48c23b019 P 55a18fe3cda74154bd8e16d6ffc31cfd: No bootstrap required, opened a new log I0918 20:39:55.272716 29997 ts_tablet_manager.cc:1085] T 7da02c564f274cf3a56af0d48c23b019 P 55a18fe3cda74154bd8e16d6ffc31cfd: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s I0918 20:39:55.274420 29999 ts_tablet_manager.cc:1113] T 7da02c564f274cf3a56af0d48c23b019 P 1c925d542dfc4c38b7f68380781223b1: Time spent starting tablet: real 0.043s user 0.006s sys 0.000s I0918 20:39:55.276787 29999 tablet_bootstrap.cc:439] T de78d398560141c6977d2344fc5c65db P 1c925d542dfc4c38b7f68380781223b1: Bootstrap starting. I0918 20:39:55.274762 29997 raft_consensus.cc:340] T 7da02c564f274cf3a56af0d48c23b019 P 55a18fe3cda74154bd8e16d6ffc31cfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } I0918 20:39:55.281213 29997 raft_consensus.cc:366] T 7da02c564f274cf3a56af0d48c23b019 P 55a18fe3cda74154bd8e16d6ffc31cfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:55.281543 29997 raft_consensus.cc:705] T 7da02c564f274cf3a56af0d48c23b019 P 55a18fe3cda74154bd8e16d6ffc31cfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a18fe3cda74154bd8e16d6ffc31cfd, State: Initialized, Role: FOLLOWER I0918 20:39:55.282464 29997 consensus_queue.cc:229] T 7da02c564f274cf3a56af0d48c23b019 P 55a18fe3cda74154bd8e16d6ffc31cfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } I0918 20:39:55.279597 29999 tablet_bootstrap.cc:590] T de78d398560141c6977d2344fc5c65db P 1c925d542dfc4c38b7f68380781223b1: No blocks or log segments found. Creating new log. I0918 20:39:55.301893 29997 ts_tablet_manager.cc:1113] T 7da02c564f274cf3a56af0d48c23b019 P 55a18fe3cda74154bd8e16d6ffc31cfd: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s I0918 20:39:55.302726 29997 tablet_bootstrap.cc:439] T cd97b4ac0e56420ebf3dc1398599f0b1 P 55a18fe3cda74154bd8e16d6ffc31cfd: Bootstrap starting. I0918 20:39:55.308282 29997 tablet_bootstrap.cc:590] T cd97b4ac0e56420ebf3dc1398599f0b1 P 55a18fe3cda74154bd8e16d6ffc31cfd: No blocks or log segments found. Creating new log. I0918 20:39:55.311833 29999 tablet_bootstrap.cc:439] T de78d398560141c6977d2344fc5c65db P 1c925d542dfc4c38b7f68380781223b1: No bootstrap required, opened a new log I0918 20:39:55.312495 29999 ts_tablet_manager.cc:1085] T de78d398560141c6977d2344fc5c65db P 1c925d542dfc4c38b7f68380781223b1: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.000s I0918 20:39:55.315126 29999 raft_consensus.cc:340] T de78d398560141c6977d2344fc5c65db P 1c925d542dfc4c38b7f68380781223b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } I0918 20:39:55.315981 29999 raft_consensus.cc:366] T de78d398560141c6977d2344fc5c65db P 1c925d542dfc4c38b7f68380781223b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:55.316228 29999 raft_consensus.cc:705] T de78d398560141c6977d2344fc5c65db P 1c925d542dfc4c38b7f68380781223b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c925d542dfc4c38b7f68380781223b1, State: Initialized, Role: FOLLOWER I0918 20:39:55.317278 29999 consensus_queue.cc:229] T de78d398560141c6977d2344fc5c65db P 1c925d542dfc4c38b7f68380781223b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } I0918 20:39:55.322276 29999 ts_tablet_manager.cc:1113] T de78d398560141c6977d2344fc5c65db P 1c925d542dfc4c38b7f68380781223b1: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s I0918 20:39:55.323983 29997 tablet_bootstrap.cc:439] T cd97b4ac0e56420ebf3dc1398599f0b1 P 55a18fe3cda74154bd8e16d6ffc31cfd: No bootstrap required, opened a new log I0918 20:39:55.325095 29997 ts_tablet_manager.cc:1085] T cd97b4ac0e56420ebf3dc1398599f0b1 P 55a18fe3cda74154bd8e16d6ffc31cfd: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s I0918 20:39:55.327450 29997 raft_consensus.cc:340] T cd97b4ac0e56420ebf3dc1398599f0b1 P 55a18fe3cda74154bd8e16d6ffc31cfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } I0918 20:39:55.328289 29997 raft_consensus.cc:366] T cd97b4ac0e56420ebf3dc1398599f0b1 P 55a18fe3cda74154bd8e16d6ffc31cfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:55.328577 29997 raft_consensus.cc:705] T cd97b4ac0e56420ebf3dc1398599f0b1 P 55a18fe3cda74154bd8e16d6ffc31cfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a18fe3cda74154bd8e16d6ffc31cfd, State: Initialized, Role: FOLLOWER I0918 20:39:55.329268 29997 consensus_queue.cc:229] T cd97b4ac0e56420ebf3dc1398599f0b1 P 55a18fe3cda74154bd8e16d6ffc31cfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } I0918 20:39:55.331225 29997 ts_tablet_manager.cc:1113] T cd97b4ac0e56420ebf3dc1398599f0b1 P 55a18fe3cda74154bd8e16d6ffc31cfd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s I0918 20:39:55.331894 30008 raft_consensus.cc:472] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:55.332361 30008 raft_consensus.cc:494] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } I0918 20:39:55.334266 30008 leader_election.cc:296] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04da295022bf4d77beae5119e3db41f5 (127.18.222.1:41197), 1c925d542dfc4c38b7f68380781223b1 (127.18.222.2:40767) I0918 20:39:55.372200 29799 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de78d398560141c6977d2344fc5c65db" candidate_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04da295022bf4d77beae5119e3db41f5" is_pre_election: true I0918 20:39:55.372995 29799 raft_consensus.cc:2350] T de78d398560141c6977d2344fc5c65db P 04da295022bf4d77beae5119e3db41f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55a18fe3cda74154bd8e16d6ffc31cfd in term 0. I0918 20:39:55.374511 29916 leader_election.cc:310] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04da295022bf4d77beae5119e3db41f5, 55a18fe3cda74154bd8e16d6ffc31cfd; no voters: I0918 20:39:55.375442 30008 raft_consensus.cc:2644] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:55.375814 30008 raft_consensus.cc:472] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:55.376060 30008 raft_consensus.cc:2890] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:55.377197 29882 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de78d398560141c6977d2344fc5c65db" candidate_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c925d542dfc4c38b7f68380781223b1" is_pre_election: true I0918 20:39:55.378016 29882 raft_consensus.cc:2350] T de78d398560141c6977d2344fc5c65db P 1c925d542dfc4c38b7f68380781223b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55a18fe3cda74154bd8e16d6ffc31cfd in term 0. I0918 20:39:55.382544 30008 raft_consensus.cc:494] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } I0918 20:39:55.384989 30008 leader_election.cc:296] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd [CANDIDATE]: Term 1 election: Requested vote from peers 04da295022bf4d77beae5119e3db41f5 (127.18.222.1:41197), 1c925d542dfc4c38b7f68380781223b1 (127.18.222.2:40767) I0918 20:39:55.386464 29799 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de78d398560141c6977d2344fc5c65db" candidate_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04da295022bf4d77beae5119e3db41f5" I0918 20:39:55.386464 29882 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de78d398560141c6977d2344fc5c65db" candidate_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c925d542dfc4c38b7f68380781223b1" I0918 20:39:55.387614 29882 raft_consensus.cc:2890] T de78d398560141c6977d2344fc5c65db P 1c925d542dfc4c38b7f68380781223b1 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:55.387245 29799 raft_consensus.cc:2890] T de78d398560141c6977d2344fc5c65db P 04da295022bf4d77beae5119e3db41f5 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:55.397796 30003 raft_consensus.cc:472] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:55.399279 30003 raft_consensus.cc:494] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } I0918 20:39:55.403954 30003 leader_election.cc:296] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55a18fe3cda74154bd8e16d6ffc31cfd (127.18.222.3:44951), 1c925d542dfc4c38b7f68380781223b1 (127.18.222.2:40767) I0918 20:39:55.404929 29799 raft_consensus.cc:2350] T de78d398560141c6977d2344fc5c65db P 04da295022bf4d77beae5119e3db41f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55a18fe3cda74154bd8e16d6ffc31cfd in term 1. I0918 20:39:55.408638 29916 leader_election.cc:310] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04da295022bf4d77beae5119e3db41f5, 55a18fe3cda74154bd8e16d6ffc31cfd; no voters: I0918 20:39:55.409685 30008 raft_consensus.cc:2644] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:55.411121 29882 raft_consensus.cc:2350] T de78d398560141c6977d2344fc5c65db P 1c925d542dfc4c38b7f68380781223b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55a18fe3cda74154bd8e16d6ffc31cfd in term 1. I0918 20:39:55.414399 30008 raft_consensus.cc:667] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd [term 1 LEADER]: Becoming Leader. State: Replica: 55a18fe3cda74154bd8e16d6ffc31cfd, State: Running, Role: LEADER I0918 20:39:55.420483 30008 consensus_queue.cc:206] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } I0918 20:39:55.432963 29709 catalog_manager.cc:4131] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd reported cstate change: term changed from 0 to 1, leader changed from to 55a18fe3cda74154bd8e16d6ffc31cfd (127.18.222.3). New cstate: current_term: 1 leader_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } health_report { overall_health: HEALTHY } } } I0918 20:39:55.434769 29953 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7da02c564f274cf3a56af0d48c23b019" candidate_uuid: "04da295022bf4d77beae5119e3db41f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" is_pre_election: true I0918 20:39:55.435617 29953 raft_consensus.cc:2350] T 7da02c564f274cf3a56af0d48c23b019 P 55a18fe3cda74154bd8e16d6ffc31cfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04da295022bf4d77beae5119e3db41f5 in term 0. I0918 20:39:55.437403 29761 leader_election.cc:310] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04da295022bf4d77beae5119e3db41f5, 55a18fe3cda74154bd8e16d6ffc31cfd; no voters: I0918 20:39:55.438493 30003 raft_consensus.cc:2644] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:55.438859 30003 raft_consensus.cc:472] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:55.439167 30003 raft_consensus.cc:2890] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:55.446897 30003 raft_consensus.cc:494] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } I0918 20:39:55.453140 30003 leader_election.cc:296] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 [CANDIDATE]: Term 1 election: Requested vote from peers 55a18fe3cda74154bd8e16d6ffc31cfd (127.18.222.3:44951), 1c925d542dfc4c38b7f68380781223b1 (127.18.222.2:40767) I0918 20:39:55.454700 29953 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7da02c564f274cf3a56af0d48c23b019" candidate_uuid: "04da295022bf4d77beae5119e3db41f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" I0918 20:39:55.456022 29953 raft_consensus.cc:2890] T 7da02c564f274cf3a56af0d48c23b019 P 55a18fe3cda74154bd8e16d6ffc31cfd [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:55.458614 29882 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7da02c564f274cf3a56af0d48c23b019" candidate_uuid: "04da295022bf4d77beae5119e3db41f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c925d542dfc4c38b7f68380781223b1" is_pre_election: true I0918 20:39:55.459424 29882 raft_consensus.cc:2350] T 7da02c564f274cf3a56af0d48c23b019 P 1c925d542dfc4c38b7f68380781223b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04da295022bf4d77beae5119e3db41f5 in term 0. I0918 20:39:55.459218 29881 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7da02c564f274cf3a56af0d48c23b019" candidate_uuid: "04da295022bf4d77beae5119e3db41f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c925d542dfc4c38b7f68380781223b1" I0918 20:39:55.460006 29881 raft_consensus.cc:2890] T 7da02c564f274cf3a56af0d48c23b019 P 1c925d542dfc4c38b7f68380781223b1 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:55.462512 29953 raft_consensus.cc:2350] T 7da02c564f274cf3a56af0d48c23b019 P 55a18fe3cda74154bd8e16d6ffc31cfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04da295022bf4d77beae5119e3db41f5 in term 1. I0918 20:39:55.468185 29761 leader_election.cc:310] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04da295022bf4d77beae5119e3db41f5, 55a18fe3cda74154bd8e16d6ffc31cfd; no voters: I0918 20:39:55.469395 29881 raft_consensus.cc:2350] T 7da02c564f274cf3a56af0d48c23b019 P 1c925d542dfc4c38b7f68380781223b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04da295022bf4d77beae5119e3db41f5 in term 1. I0918 20:39:55.472410 30003 raft_consensus.cc:2644] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:55.488971 30003 raft_consensus.cc:667] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 [term 1 LEADER]: Becoming Leader. State: Replica: 04da295022bf4d77beae5119e3db41f5, State: Running, Role: LEADER I0918 20:39:55.491169 30003 consensus_queue.cc:206] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } I0918 20:39:55.508457 29709 catalog_manager.cc:4131] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 reported cstate change: term changed from 0 to 1, leader changed from to 04da295022bf4d77beae5119e3db41f5 (127.18.222.1). New cstate: current_term: 1 leader_uuid: "04da295022bf4d77beae5119e3db41f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } health_report { overall_health: UNKNOWN } } } I0918 20:39:55.526453 30004 raft_consensus.cc:472] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:55.527115 30004 raft_consensus.cc:494] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } I0918 20:39:55.529240 30004 leader_election.cc:296] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04da295022bf4d77beae5119e3db41f5 (127.18.222.1:41197), 55a18fe3cda74154bd8e16d6ffc31cfd (127.18.222.3:44951) I0918 20:39:55.579882 29799 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "cd97b4ac0e56420ebf3dc1398599f0b1" candidate_uuid: "1c925d542dfc4c38b7f68380781223b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04da295022bf4d77beae5119e3db41f5" is_pre_election: true I0918 20:39:55.580610 29799 raft_consensus.cc:2350] T cd97b4ac0e56420ebf3dc1398599f0b1 P 04da295022bf4d77beae5119e3db41f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c925d542dfc4c38b7f68380781223b1 in term 0. I0918 20:39:55.582001 29836 leader_election.cc:310] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04da295022bf4d77beae5119e3db41f5, 1c925d542dfc4c38b7f68380781223b1; no voters: I0918 20:39:55.582947 30004 raft_consensus.cc:2644] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:55.583267 30004 raft_consensus.cc:472] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:55.583537 30004 raft_consensus.cc:2890] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:55.584666 29953 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "cd97b4ac0e56420ebf3dc1398599f0b1" candidate_uuid: "1c925d542dfc4c38b7f68380781223b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" is_pre_election: true I0918 20:39:55.585450 29953 raft_consensus.cc:2350] T cd97b4ac0e56420ebf3dc1398599f0b1 P 55a18fe3cda74154bd8e16d6ffc31cfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c925d542dfc4c38b7f68380781223b1 in term 0. I0918 20:39:55.595088 30004 raft_consensus.cc:494] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } I0918 20:39:55.597826 30004 leader_election.cc:296] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 [CANDIDATE]: Term 1 election: Requested vote from peers 04da295022bf4d77beae5119e3db41f5 (127.18.222.1:41197), 55a18fe3cda74154bd8e16d6ffc31cfd (127.18.222.3:44951) I0918 20:39:55.597738 29799 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "cd97b4ac0e56420ebf3dc1398599f0b1" candidate_uuid: "1c925d542dfc4c38b7f68380781223b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04da295022bf4d77beae5119e3db41f5" I0918 20:39:55.598508 29799 raft_consensus.cc:2890] T cd97b4ac0e56420ebf3dc1398599f0b1 P 04da295022bf4d77beae5119e3db41f5 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:55.598747 29953 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "cd97b4ac0e56420ebf3dc1398599f0b1" candidate_uuid: "1c925d542dfc4c38b7f68380781223b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" I0918 20:39:55.599391 29953 raft_consensus.cc:2890] T cd97b4ac0e56420ebf3dc1398599f0b1 P 55a18fe3cda74154bd8e16d6ffc31cfd [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:55.605967 29953 raft_consensus.cc:2350] T cd97b4ac0e56420ebf3dc1398599f0b1 P 55a18fe3cda74154bd8e16d6ffc31cfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c925d542dfc4c38b7f68380781223b1 in term 1. I0918 20:39:55.607288 29835 leader_election.cc:310] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c925d542dfc4c38b7f68380781223b1, 55a18fe3cda74154bd8e16d6ffc31cfd; no voters: I0918 20:39:55.608587 30004 raft_consensus.cc:2644] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:55.609088 29799 raft_consensus.cc:2350] T cd97b4ac0e56420ebf3dc1398599f0b1 P 04da295022bf4d77beae5119e3db41f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c925d542dfc4c38b7f68380781223b1 in term 1. I0918 20:39:55.610623 30004 raft_consensus.cc:667] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 [term 1 LEADER]: Becoming Leader. State: Replica: 1c925d542dfc4c38b7f68380781223b1, State: Running, Role: LEADER I0918 20:39:55.611378 30004 consensus_queue.cc:206] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } } I0918 20:39:55.620724 29711 catalog_manager.cc:4131] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 reported cstate change: term changed from 0 to 1, leader changed from to 1c925d542dfc4c38b7f68380781223b1 (127.18.222.2). New cstate: current_term: 1 leader_uuid: "1c925d542dfc4c38b7f68380781223b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 } health_report { overall_health: UNKNOWN } } } I0918 20:39:55.851135 30008 consensus_queue.cc:984] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:55.872967 30008 consensus_queue.cc:984] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:55.966404 30003 consensus_queue.cc:984] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:55.993818 30003 consensus_queue.cc:984] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c925d542dfc4c38b7f68380781223b1" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:56.023201 30004 consensus_queue.cc:984] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55a18fe3cda74154bd8e16d6ffc31cfd" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 44951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:56.041854 30030 consensus_queue.cc:984] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04da295022bf4d77beae5119e3db41f5" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 41197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:56.503337 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:56.575485 29761 messenger.cc:381] Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer W0918 20:39:56.578316 29836 consensus_peers.cc:458] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 -> Peer 04da295022bf4d77beae5119e3db41f5 (127.18.222.1:41197): Couldn't send request to peer 04da295022bf4d77beae5119e3db41f5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry. I0918 20:39:56.589586 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:56.590060 19320 tablet_replica.cc:273] T de78d398560141c6977d2344fc5c65db P 04da295022bf4d77beae5119e3db41f5: stopping tablet replica I0918 20:39:56.590466 19320 raft_consensus.cc:2147] T de78d398560141c6977d2344fc5c65db P 04da295022bf4d77beae5119e3db41f5 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:56.590785 19320 raft_consensus.cc:2174] T de78d398560141c6977d2344fc5c65db P 04da295022bf4d77beae5119e3db41f5 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:56.594080 19320 tablet_replica.cc:273] T cd97b4ac0e56420ebf3dc1398599f0b1 P 04da295022bf4d77beae5119e3db41f5: stopping tablet replica I0918 20:39:56.594748 19320 raft_consensus.cc:2147] T cd97b4ac0e56420ebf3dc1398599f0b1 P 04da295022bf4d77beae5119e3db41f5 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:56.595202 19320 raft_consensus.cc:2174] T cd97b4ac0e56420ebf3dc1398599f0b1 P 04da295022bf4d77beae5119e3db41f5 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:56.600262 19320 tablet_replica.cc:273] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5: stopping tablet replica I0918 20:39:56.600845 19320 raft_consensus.cc:2147] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:56.601570 19320 raft_consensus.cc:2174] T 7da02c564f274cf3a56af0d48c23b019 P 04da295022bf4d77beae5119e3db41f5 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:56.632372 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:56.632575 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:56.632922 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:56.649379 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:56.692987 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:56.693351 19320 tablet_replica.cc:273] T de78d398560141c6977d2344fc5c65db P 1c925d542dfc4c38b7f68380781223b1: stopping tablet replica I0918 20:39:56.693843 19320 raft_consensus.cc:2147] T de78d398560141c6977d2344fc5c65db P 1c925d542dfc4c38b7f68380781223b1 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:56.694257 19320 raft_consensus.cc:2174] T de78d398560141c6977d2344fc5c65db P 1c925d542dfc4c38b7f68380781223b1 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:56.697227 19320 tablet_replica.cc:273] T 7da02c564f274cf3a56af0d48c23b019 P 1c925d542dfc4c38b7f68380781223b1: stopping tablet replica I0918 20:39:56.697693 19320 raft_consensus.cc:2147] T 7da02c564f274cf3a56af0d48c23b019 P 1c925d542dfc4c38b7f68380781223b1 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:56.697994 19320 raft_consensus.cc:2174] T 7da02c564f274cf3a56af0d48c23b019 P 1c925d542dfc4c38b7f68380781223b1 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:56.701802 19320 tablet_replica.cc:273] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1: stopping tablet replica I0918 20:39:56.702337 19320 raft_consensus.cc:2147] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:56.703220 19320 raft_consensus.cc:2174] T cd97b4ac0e56420ebf3dc1398599f0b1 P 1c925d542dfc4c38b7f68380781223b1 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:56.726994 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:56.727195 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:56.727577 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:56.740234 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:56.775936 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:56.777145 19320 tablet_replica.cc:273] T cd97b4ac0e56420ebf3dc1398599f0b1 P 55a18fe3cda74154bd8e16d6ffc31cfd: stopping tablet replica I0918 20:39:56.777689 19320 raft_consensus.cc:2147] T cd97b4ac0e56420ebf3dc1398599f0b1 P 55a18fe3cda74154bd8e16d6ffc31cfd [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:56.778002 19320 raft_consensus.cc:2174] T cd97b4ac0e56420ebf3dc1398599f0b1 P 55a18fe3cda74154bd8e16d6ffc31cfd [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:56.782155 19320 tablet_replica.cc:273] T 7da02c564f274cf3a56af0d48c23b019 P 55a18fe3cda74154bd8e16d6ffc31cfd: stopping tablet replica I0918 20:39:56.782519 19320 raft_consensus.cc:2147] T 7da02c564f274cf3a56af0d48c23b019 P 55a18fe3cda74154bd8e16d6ffc31cfd [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:56.782788 19320 raft_consensus.cc:2174] T 7da02c564f274cf3a56af0d48c23b019 P 55a18fe3cda74154bd8e16d6ffc31cfd [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:56.785099 19320 tablet_replica.cc:273] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd: stopping tablet replica I0918 20:39:56.785496 19320 raft_consensus.cc:2147] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:56.786140 19320 raft_consensus.cc:2174] T de78d398560141c6977d2344fc5c65db P 55a18fe3cda74154bd8e16d6ffc31cfd [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:56.809381 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:56.809583 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:56.809978 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:39:56.819855 19320 master.cc:247] Master@127.18.222.62:45907 shutting down... I0918 20:39:56.836371 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:56.836757 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:56.837002 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P f42f36621b204e808330eb34ce6c6842: stopping tablet replica I0918 20:39:56.855237 19320 master.cc:258] Master@127.18.222.62:45907 shutdown complete. I0918 20:39:56.864476 19320 master.cc:247] Master@127.18.222.61:44027 shutting down... I0918 20:39:56.884598 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:56.885001 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:56.885239 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 9524c898626646999a750e57a583d010: stopping tablet replica I0918 20:39:56.903220 19320 master.cc:258] Master@127.18.222.61:44027 shutdown complete. I0918 20:39:56.913338 19320 master.cc:247] Master@127.18.222.60:41777 shutting down... I0918 20:39:56.930290 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [term 1 LEADER]: Raft consensus shutting down. I0918 20:39:56.931056 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:56.931318 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 510fdaab3c2f457bb828ac464cf414ae: stopping tablet replica I0918 20:39:56.950037 19320 master.cc:258] Master@127.18.222.60:41777 shutdown complete. [ OK ] RemoteKsckTest.TestTabletFiltersNoMatch (3330 ms) [ RUN ] RemoteKsckTest.TestTabletFilters I0918 20:39:56.980896 19320 test_util.cc:212] Using random seed: -687098238 I0918 20:39:56.981648 19320 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.18.222.62:43789,127.18.222.61:45529,127.18.222.60:34087 I0918 20:39:56.994504 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 175592us I0918 20:39:56.994838 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:56.994977 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:56.995175 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:56.995287 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:56.996290 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/master-0-root/instance: uuid: "09058351a47546a7a1d3065bb052c0a7" format_stamp: "Formatted at 2019-09-18 20:39:56 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:57.005018 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s I0918 20:39:57.006515 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:57.013329 19320 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0918 20:39:57.013510 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/master-0-root uuid: "09058351a47546a7a1d3065bb052c0a7" format_stamp: "Formatted at 2019-09-18 20:39:56 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:57.013738 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:57.056708 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.62:43789 I0918 20:39:57.058360 19320 webserver.cc:374] Webserver started at http://127.18.222.62:41873/ using document root and password file I0918 20:39:57.060134 30134 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:57.064903 30134 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:57.077427 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 176092us I0918 20:39:57.077901 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:57.078188 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:57.078416 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:57.078544 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:57.079751 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/master-1-root/instance: uuid: "de19abd28eca4f258b52bf7ba47a80a6" format_stamp: "Formatted at 2019-09-18 20:39:57 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:57.079506 30134 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:57.081893 30134 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:45529: Network error: Client connection negotiation failed: client connection to 127.18.222.61:45529: connect: Connection refused (error 111) I0918 20:39:57.090248 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.010s sys 0.000s I0918 20:39:57.092047 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:57.100278 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0918 20:39:57.100486 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/master-1-root uuid: "de19abd28eca4f258b52bf7ba47a80a6" format_stamp: "Formatted at 2019-09-18 20:39:57 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:57.100783 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/master-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:57.123319 30134 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } attempt: 1 W0918 20:39:57.125547 30134 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:45529: Network error: Client connection negotiation failed: client connection to 127.18.222.61:45529: connect: Connection refused (error 111) I0918 20:39:57.171036 30134 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } attempt: 2 W0918 20:39:57.173223 30134 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.61:45529: Network error: Client connection negotiation failed: client connection to 127.18.222.61:45529: connect: Connection refused (error 111) I0918 20:39:57.183440 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.61:45529 I0918 20:39:57.185633 19320 webserver.cc:374] Webserver started at http://127.18.222.61:35721/ using document root and password file I0918 20:39:57.187985 30191 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:57.195098 30191 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:57.209610 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 176092us I0918 20:39:57.210186 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:57.210391 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:57.210606 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:57.210732 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:57.211205 30191 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:57.212121 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/master-2-root/instance: uuid: "e3d8bb9e6e304aaea1273fc041c2f160" format_stamp: "Formatted at 2019-09-18 20:39:57 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:57.227057 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.014s user 0.010s sys 0.003s I0918 20:39:57.227308 30191 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:57.229395 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 W0918 20:39:57.229612 30191 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:34087: Network error: Client connection negotiation failed: client connection to 127.18.222.60:34087: connect: Connection refused (error 111) I0918 20:39:57.237653 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:39:57.237890 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/master-2-root uuid: "e3d8bb9e6e304aaea1273fc041c2f160" format_stamp: "Formatted at 2019-09-18 20:39:57 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:57.238163 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/master-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:57.249650 30134 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } attempt: 3 I0918 20:39:57.261893 30134 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:57.263768 30134 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:34087: Network error: Client connection negotiation failed: client connection to 127.18.222.60:34087: connect: Connection refused (error 111) I0918 20:39:57.269150 30191 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } attempt: 1 W0918 20:39:57.271375 30191 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:34087: Network error: Client connection negotiation failed: client connection to 127.18.222.60:34087: connect: Connection refused (error 111) I0918 20:39:57.289180 30134 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } attempt: 1 W0918 20:39:57.291788 30134 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.18.222.60:34087: Network error: Client connection negotiation failed: client connection to 127.18.222.60:34087: connect: Connection refused (error 111) I0918 20:39:57.308881 30191 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } attempt: 2 I0918 20:39:57.310503 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.60:34087 I0918 20:39:57.318428 19320 webserver.cc:374] Webserver started at http://127.18.222.60:38265/ using document root and password file I0918 20:39:57.320230 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0918 20:39:57.321058 30249 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:57.327438 30249 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:57.332245 30191 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6: Bootstrap starting. I0918 20:39:57.335150 30191 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6: No blocks or log segments found. Creating new log. I0918 20:39:57.339440 30191 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6: No bootstrap required, opened a new log I0918 20:39:57.341202 30134 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } attempt: 2 I0918 20:39:57.341387 30191 raft_consensus.cc:340] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } I0918 20:39:57.342154 30191 raft_consensus.cc:366] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:57.342367 30191 raft_consensus.cc:705] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de19abd28eca4f258b52bf7ba47a80a6, State: Initialized, Role: FOLLOWER I0918 20:39:57.343014 30191 consensus_queue.cc:229] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } I0918 20:39:57.345235 30191 sys_catalog.cc:340] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } } I0918 20:39:57.345598 30252 sys_catalog.cc:340] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } } I0918 20:39:57.346068 30191 sys_catalog.cc:343] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:57.346525 30252 sys_catalog.cc:343] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:57.347473 30191 sys_catalog.cc:428] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:57.350317 30249 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } has no permanent_uuid. Determining permanent_uuid... W0918 20:39:57.351852 30253 catalog_manager.cc:1155] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:57.352536 30253 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:57.352613 30252 raft_consensus.cc:472] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:57.353255 30252 raft_consensus.cc:494] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } I0918 20:39:57.355944 30252 leader_election.cc:296] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09058351a47546a7a1d3065bb052c0a7 (127.18.222.62:43789), e3d8bb9e6e304aaea1273fc041c2f160 (127.18.222.60:34087) I0918 20:39:57.365027 30249 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } has no permanent_uuid. Determining permanent_uuid... I0918 20:39:57.366240 30134 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7: Bootstrap starting. I0918 20:39:57.371206 30134 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7: No blocks or log segments found. Creating new log. I0918 20:39:57.375150 30134 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7: No bootstrap required, opened a new log I0918 20:39:57.377107 30134 raft_consensus.cc:340] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } I0918 20:39:57.377846 30134 raft_consensus.cc:366] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:57.378095 30134 raft_consensus.cc:705] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09058351a47546a7a1d3065bb052c0a7, State: Initialized, Role: FOLLOWER I0918 20:39:57.378764 30134 consensus_queue.cc:229] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } I0918 20:39:57.380781 30134 sys_catalog.cc:340] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } } I0918 20:39:57.381527 30134 sys_catalog.cc:343] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:57.380995 30255 sys_catalog.cc:340] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } } I0918 20:39:57.381980 30255 sys_catalog.cc:343] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:57.382696 30134 sys_catalog.cc:428] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:57.383778 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 I0918 20:39:57.384163 19320 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 I0918 20:39:57.383627 30111 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de19abd28eca4f258b52bf7ba47a80a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09058351a47546a7a1d3065bb052c0a7" is_pre_election: true I0918 20:39:57.384673 30111 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de19abd28eca4f258b52bf7ba47a80a6 in term 0. I0918 20:39:57.385938 30140 leader_election.cc:310] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09058351a47546a7a1d3065bb052c0a7, de19abd28eca4f258b52bf7ba47a80a6; no voters: I0918 20:39:57.386878 30252 raft_consensus.cc:2644] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:57.387168 30252 raft_consensus.cc:472] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:57.387531 30252 raft_consensus.cc:2890] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:57.385874 30249 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160: Bootstrap starting. I0918 20:39:57.391516 30252 raft_consensus.cc:494] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } W0918 20:39:57.392841 30256 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0918 20:39:57.393198 30256 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0918 20:39:57.391506 30249 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160: No blocks or log segments found. Creating new log. I0918 20:39:57.393399 30252 leader_election.cc:296] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [CANDIDATE]: Term 1 election: Requested vote from peers 09058351a47546a7a1d3065bb052c0a7 (127.18.222.62:43789), e3d8bb9e6e304aaea1273fc041c2f160 (127.18.222.60:34087) I0918 20:39:57.393817 30111 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de19abd28eca4f258b52bf7ba47a80a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09058351a47546a7a1d3065bb052c0a7" I0918 20:39:57.394485 30111 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:57.397644 30249 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160: No bootstrap required, opened a new log I0918 20:39:57.400266 30249 raft_consensus.cc:340] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } I0918 20:39:57.400996 30249 raft_consensus.cc:366] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:57.400979 30111 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de19abd28eca4f258b52bf7ba47a80a6 in term 1. I0918 20:39:57.401248 30249 raft_consensus.cc:705] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3d8bb9e6e304aaea1273fc041c2f160, State: Initialized, Role: FOLLOWER I0918 20:39:57.401988 30249 consensus_queue.cc:229] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } I0918 20:39:57.402492 30140 leader_election.cc:310] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09058351a47546a7a1d3065bb052c0a7, de19abd28eca4f258b52bf7ba47a80a6; no voters: I0918 20:39:57.403807 30252 raft_consensus.cc:2644] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:57.404081 30249 sys_catalog.cc:340] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } } I0918 20:39:57.404341 30258 sys_catalog.cc:340] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } } I0918 20:39:57.404985 30249 sys_catalog.cc:343] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:57.405184 30258 sys_catalog.cc:343] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:57.405895 30252 raft_consensus.cc:667] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [term 1 LEADER]: Becoming Leader. State: Replica: de19abd28eca4f258b52bf7ba47a80a6, State: Running, Role: LEADER I0918 20:39:57.406716 30249 sys_catalog.cc:428] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160 [sys.catalog]: configured and running, proceeding with master startup. I0918 20:39:57.406826 30224 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de19abd28eca4f258b52bf7ba47a80a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" I0918 20:39:57.406705 30252 consensus_queue.cc:206] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } I0918 20:39:57.407843 30224 raft_consensus.cc:2890] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160 [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:57.407976 30260 catalog_manager.cc:569] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0918 20:39:57.410450 30225 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de19abd28eca4f258b52bf7ba47a80a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" is_pre_election: true I0918 20:39:57.411543 30259 sys_catalog.cc:340] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de19abd28eca4f258b52bf7ba47a80a6. Latest consensus state: current_term: 1 leader_uuid: "de19abd28eca4f258b52bf7ba47a80a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } } I0918 20:39:57.412369 30259 sys_catalog.cc:343] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [sys.catalog]: This master's current role is: LEADER I0918 20:39:57.416093 30262 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0918 20:39:57.416854 30224 raft_consensus.cc:2350] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de19abd28eca4f258b52bf7ba47a80a6 in term 1. I0918 20:39:57.417932 30225 raft_consensus.cc:2325] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de19abd28eca4f258b52bf7ba47a80a6 for term 1 because replica is already servicing an update from a current leader or another vote. I0918 20:39:57.419265 30262 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0918 20:39:57.428277 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 176092us I0918 20:39:57.428684 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:57.428846 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:57.429069 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:57.429190 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:57.430205 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/ts-0-root/instance: uuid: "aa62fa80d8be490fbb5a2d10eef5235f" format_stamp: "Formatted at 2019-09-18 20:39:57 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:57.440402 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.011s sys 0.001s I0918 20:39:57.442018 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:57.450279 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0918 20:39:57.450496 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/ts-0-root uuid: "aa62fa80d8be490fbb5a2d10eef5235f" format_stamp: "Formatted at 2019-09-18 20:39:57 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:57.450750 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:57.456640 30111 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7 [term 1 FOLLOWER]: Refusing update from remote peer de19abd28eca4f258b52bf7ba47a80a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:57.456737 30225 raft_consensus.cc:1184] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160 [term 1 FOLLOWER]: Refusing update from remote peer de19abd28eca4f258b52bf7ba47a80a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0918 20:39:57.458264 30259 consensus_queue.cc:984] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:57.459013 30252 consensus_queue.cc:984] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:57.465734 30255 sys_catalog.cc:340] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de19abd28eca4f258b52bf7ba47a80a6. Latest consensus state: current_term: 1 leader_uuid: "de19abd28eca4f258b52bf7ba47a80a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } } I0918 20:39:57.466629 30255 sys_catalog.cc:343] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:57.474529 30255 sys_catalog.cc:340] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "de19abd28eca4f258b52bf7ba47a80a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } } I0918 20:39:57.475109 30255 sys_catalog.cc:343] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:57.478448 30252 sys_catalog.cc:340] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "de19abd28eca4f258b52bf7ba47a80a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } } I0918 20:39:57.479851 30252 sys_catalog.cc:343] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [sys.catalog]: This master's current role is: LEADER I0918 20:39:57.485446 30258 sys_catalog.cc:340] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de19abd28eca4f258b52bf7ba47a80a6. Latest consensus state: current_term: 1 leader_uuid: "de19abd28eca4f258b52bf7ba47a80a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } } I0918 20:39:57.486605 30258 sys_catalog.cc:343] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:57.486734 30262 catalog_manager.cc:981] Generated new certificate authority record I0918 20:39:57.489655 30262 catalog_manager.cc:1106] Loading token signing keys... I0918 20:39:57.490128 30276 sys_catalog.cc:340] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "de19abd28eca4f258b52bf7ba47a80a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } } I0918 20:39:57.491197 30276 sys_catalog.cc:343] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [sys.catalog]: This master's current role is: LEADER I0918 20:39:57.500196 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:57.500576 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:57.500438 30258 sys_catalog.cc:340] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "de19abd28eca4f258b52bf7ba47a80a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09058351a47546a7a1d3065bb052c0a7" member_type: VOTER last_known_addr { host: "127.18.222.62" port: 43789 } } peers { permanent_uuid: "de19abd28eca4f258b52bf7ba47a80a6" member_type: VOTER last_known_addr { host: "127.18.222.61" port: 45529 } } peers { permanent_uuid: "e3d8bb9e6e304aaea1273fc041c2f160" member_type: VOTER last_known_addr { host: "127.18.222.60" port: 34087 } } } I0918 20:39:57.501485 30258 sys_catalog.cc:343] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160 [sys.catalog]: This master's current role is: FOLLOWER I0918 20:39:57.517087 30262 catalog_manager.cc:4446] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6: Generated new TSK 0 I0918 20:39:57.518061 30262 catalog_manager.cc:1116] Initializing in-progress tserver states... I0918 20:39:57.549068 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.1:34719 I0918 20:39:57.551071 19320 webserver.cc:374] Webserver started at http://127.18.222.1:33221/ using document root and password file I0918 20:39:57.578943 30337 heartbeater.cc:322] Connected to a master server at 127.18.222.60:34087 I0918 20:39:57.579017 30336 heartbeater.cc:322] Connected to a master server at 127.18.222.61:45529 I0918 20:39:57.579396 30337 heartbeater.cc:406] Registering TS with master... I0918 20:39:57.579485 30336 heartbeater.cc:406] Registering TS with master... I0918 20:39:57.580077 30337 heartbeater.cc:455] Master 127.18.222.60:34087 requested a full tablet report, sending... I0918 20:39:57.580224 30336 heartbeater.cc:455] Master 127.18.222.61:45529 requested a full tablet report, sending... I0918 20:39:57.582306 30158 ts_manager.cc:186] Registered new tserver with Master: aa62fa80d8be490fbb5a2d10eef5235f (127.18.222.1:34719) I0918 20:39:57.582937 30215 ts_manager.cc:186] Registered new tserver with Master: aa62fa80d8be490fbb5a2d10eef5235f (127.18.222.1:34719) I0918 20:39:57.585347 30158 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34590 I0918 20:39:57.587918 30338 heartbeater.cc:322] Connected to a master server at 127.18.222.62:43789 I0918 20:39:57.588177 30338 heartbeater.cc:406] Registering TS with master... I0918 20:39:57.588649 30338 heartbeater.cc:455] Master 127.18.222.62:43789 requested a full tablet report, sending... I0918 20:39:57.590176 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 176092us I0918 20:39:57.590543 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:57.590701 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:57.590610 30101 ts_manager.cc:186] Registered new tserver with Master: aa62fa80d8be490fbb5a2d10eef5235f (127.18.222.1:34719) I0918 20:39:57.590955 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:57.591161 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:57.592316 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/ts-1-root/instance: uuid: "04a5db21380e475189ae3c20003c489f" format_stamp: "Formatted at 2019-09-18 20:39:57 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:57.598966 30336 heartbeater.cc:447] Master 127.18.222.61:45529 was elected leader, sending a full tablet report... I0918 20:39:57.603044 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.007s sys 0.003s I0918 20:39:57.604809 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:57.613207 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I0918 20:39:57.613440 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/ts-1-root uuid: "04a5db21380e475189ae3c20003c489f" format_stamp: "Formatted at 2019-09-18 20:39:57 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:57.613687 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:57.636095 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:57.636313 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:57.683928 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.2:38355 I0918 20:39:57.685992 19320 webserver.cc:374] Webserver started at http://127.18.222.2:45575/ using document root and password file I0918 20:39:57.718786 30407 heartbeater.cc:322] Connected to a master server at 127.18.222.61:45529 I0918 20:39:57.719177 30407 heartbeater.cc:406] Registering TS with master... I0918 20:39:57.719676 30407 heartbeater.cc:455] Master 127.18.222.61:45529 requested a full tablet report, sending... I0918 20:39:57.721554 30158 ts_manager.cc:186] Registered new tserver with Master: 04a5db21380e475189ae3c20003c489f (127.18.222.2:38355) I0918 20:39:57.724021 30158 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34596 I0918 20:39:57.725101 30408 heartbeater.cc:322] Connected to a master server at 127.18.222.60:34087 I0918 20:39:57.725425 30408 heartbeater.cc:406] Registering TS with master... I0918 20:39:57.726269 30408 heartbeater.cc:455] Master 127.18.222.60:34087 requested a full tablet report, sending... I0918 20:39:57.728459 30215 ts_manager.cc:186] Registered new tserver with Master: 04a5db21380e475189ae3c20003c489f (127.18.222.2:38355) I0918 20:39:57.733530 19320 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 176092us I0918 20:39:57.733922 19320 fs_manager.cc:263] Metadata directory not provided I0918 20:39:57.734102 19320 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0918 20:39:57.734387 19320 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0918 20:39:57.734537 19320 server_base.cc:453] Attempting to create new FS layout instead I0918 20:39:57.734925 30410 heartbeater.cc:322] Connected to a master server at 127.18.222.62:43789 I0918 20:39:57.735363 30410 heartbeater.cc:406] Registering TS with master... I0918 20:39:57.735837 30410 heartbeater.cc:455] Master 127.18.222.62:43789 requested a full tablet report, sending... I0918 20:39:57.735800 19320 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/ts-2-root/instance: uuid: "f56d73728210458c871a73e1eaf928ac" format_stamp: "Formatted at 2019-09-18 20:39:57 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:57.737481 30407 heartbeater.cc:447] Master 127.18.222.61:45529 was elected leader, sending a full tablet report... I0918 20:39:57.737984 30101 ts_manager.cc:186] Registered new tserver with Master: 04a5db21380e475189ae3c20003c489f (127.18.222.2:38355) I0918 20:39:57.749330 19320 fs_manager.cc:503] Time spent creating directory manager: real 0.013s user 0.012s sys 0.001s I0918 20:39:57.752041 19320 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0918 20:39:57.762405 19320 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0918 20:39:57.762712 19320 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/ts-2-root uuid: "f56d73728210458c871a73e1eaf928ac" format_stamp: "Formatted at 2019-09-18 20:39:57 on dist-test-slave-dist-test-slave-85lk" I0918 20:39:57.763048 19320 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-task6lCVqo/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1568839089958813-19320/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0918 20:39:57.842442 19320 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0918 20:39:57.842661 19320 ts_tablet_manager.cc:403] Registered 0 tablets I0918 20:39:57.909518 19320 rpc_server.cc:205] RPC server started. Bound to: 127.18.222.3:42979 I0918 20:39:57.912317 19320 webserver.cc:374] Webserver started at http://127.18.222.3:45647/ using document root and password file I0918 20:39:57.983958 30478 heartbeater.cc:322] Connected to a master server at 127.18.222.61:45529 I0918 20:39:57.984365 30478 heartbeater.cc:406] Registering TS with master... I0918 20:39:57.985103 30478 heartbeater.cc:455] Master 127.18.222.61:45529 requested a full tablet report, sending... I0918 20:39:57.986114 30479 heartbeater.cc:322] Connected to a master server at 127.18.222.60:34087 I0918 20:39:57.986671 30479 heartbeater.cc:406] Registering TS with master... I0918 20:39:57.987262 30479 heartbeater.cc:455] Master 127.18.222.60:34087 requested a full tablet report, sending... I0918 20:39:57.987164 30157 ts_manager.cc:186] Registered new tserver with Master: f56d73728210458c871a73e1eaf928ac (127.18.222.3:42979) I0918 20:39:57.989609 30214 ts_manager.cc:186] Registered new tserver with Master: f56d73728210458c871a73e1eaf928ac (127.18.222.3:42979) I0918 20:39:57.989908 30157 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34602 I0918 20:39:57.992648 30480 heartbeater.cc:322] Connected to a master server at 127.18.222.62:43789 I0918 20:39:57.992952 30480 heartbeater.cc:406] Registering TS with master... I0918 20:39:57.993607 30480 heartbeater.cc:455] Master 127.18.222.62:43789 requested a full tablet report, sending... I0918 20:39:57.996140 30100 ts_manager.cc:186] Registered new tserver with Master: f56d73728210458c871a73e1eaf928ac (127.18.222.3:42979) I0918 20:39:57.996959 19320 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.035315855s I0918 20:39:58.006496 30478 heartbeater.cc:447] Master 127.18.222.61:45529 was elected leader, sending a full tablet report... I0918 20:39:58.078361 30156 catalog_manager.cc:1451] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34612: name: "ksck-test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\n\000\000\000\004\001\024\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } W0918 20:39:58.080735 30156 catalog_manager.cc:1616] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0918 20:39:58.287917 30445 tablet_service.cc:1022] Processing CreateTablet for tablet c95a78d505f64699b76403bb22206589 (table=ksck-test-table [id=ae1c6d72fc7c49b8ad3a7f48d1ac71f9]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:58.289248 30445 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet c95a78d505f64699b76403bb22206589. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:58.299937 30444 tablet_service.cc:1022] Processing CreateTablet for tablet f2533ad3d1db420c890b4ec86118331a (table=ksck-test-table [id=ae1c6d72fc7c49b8ad3a7f48d1ac71f9]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:58.301218 30444 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f2533ad3d1db420c890b4ec86118331a. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:58.302975 30374 tablet_service.cc:1022] Processing CreateTablet for tablet c95a78d505f64699b76403bb22206589 (table=ksck-test-table [id=ae1c6d72fc7c49b8ad3a7f48d1ac71f9]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:58.304056 30374 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet c95a78d505f64699b76403bb22206589. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:58.308328 30302 tablet_service.cc:1022] Processing CreateTablet for tablet f2533ad3d1db420c890b4ec86118331a (table=ksck-test-table [id=ae1c6d72fc7c49b8ad3a7f48d1ac71f9]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:58.309566 30302 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f2533ad3d1db420c890b4ec86118331a. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:58.310454 30301 tablet_service.cc:1022] Processing CreateTablet for tablet 6557cb6da784488b900635b19d9df9c7 (table=ksck-test-table [id=ae1c6d72fc7c49b8ad3a7f48d1ac71f9]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:58.315884 30373 tablet_service.cc:1022] Processing CreateTablet for tablet f2533ad3d1db420c890b4ec86118331a (table=ksck-test-table [id=ae1c6d72fc7c49b8ad3a7f48d1ac71f9]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 I0918 20:39:58.311287 30303 tablet_service.cc:1022] Processing CreateTablet for tablet c95a78d505f64699b76403bb22206589 (table=ksck-test-table [id=ae1c6d72fc7c49b8ad3a7f48d1ac71f9]), partition=RANGE (key) PARTITION VALUES < 10 I0918 20:39:58.316344 30301 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 6557cb6da784488b900635b19d9df9c7. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:58.311916 30443 tablet_service.cc:1022] Processing CreateTablet for tablet 6557cb6da784488b900635b19d9df9c7 (table=ksck-test-table [id=ae1c6d72fc7c49b8ad3a7f48d1ac71f9]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:58.317751 30443 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 6557cb6da784488b900635b19d9df9c7. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:58.316995 30373 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f2533ad3d1db420c890b4ec86118331a. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:58.323978 30303 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet c95a78d505f64699b76403bb22206589. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:58.333719 30372 tablet_service.cc:1022] Processing CreateTablet for tablet 6557cb6da784488b900635b19d9df9c7 (table=ksck-test-table [id=ae1c6d72fc7c49b8ad3a7f48d1ac71f9]), partition=RANGE (key) PARTITION VALUES >= 20 I0918 20:39:58.335189 30372 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 6557cb6da784488b900635b19d9df9c7. 1 dirs total, 0 dirs full, 0 dirs failed I0918 20:39:58.401243 30256 catalog_manager.cc:1153] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7: acquiring CA information for follower catalog manager: success I0918 20:39:58.402809 30507 tablet_bootstrap.cc:439] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac: Bootstrap starting. I0918 20:39:58.408946 30256 catalog_manager.cc:1181] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:58.426245 30260 catalog_manager.cc:1153] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160: acquiring CA information for follower catalog manager: success I0918 20:39:58.411675 30507 tablet_bootstrap.cc:590] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac: No blocks or log segments found. Creating new log. I0918 20:39:58.434537 30508 tablet_bootstrap.cc:439] T f2533ad3d1db420c890b4ec86118331a P aa62fa80d8be490fbb5a2d10eef5235f: Bootstrap starting. I0918 20:39:58.443223 30508 tablet_bootstrap.cc:590] T f2533ad3d1db420c890b4ec86118331a P aa62fa80d8be490fbb5a2d10eef5235f: No blocks or log segments found. Creating new log. I0918 20:39:58.444490 30260 catalog_manager.cc:1181] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0918 20:39:58.462303 30509 tablet_bootstrap.cc:439] T f2533ad3d1db420c890b4ec86118331a P 04a5db21380e475189ae3c20003c489f: Bootstrap starting. I0918 20:39:58.469700 30509 tablet_bootstrap.cc:590] T f2533ad3d1db420c890b4ec86118331a P 04a5db21380e475189ae3c20003c489f: No blocks or log segments found. Creating new log. I0918 20:39:58.475162 30508 tablet_bootstrap.cc:439] T f2533ad3d1db420c890b4ec86118331a P aa62fa80d8be490fbb5a2d10eef5235f: No bootstrap required, opened a new log I0918 20:39:58.475752 30508 ts_tablet_manager.cc:1085] T f2533ad3d1db420c890b4ec86118331a P aa62fa80d8be490fbb5a2d10eef5235f: Time spent bootstrapping tablet: real 0.041s user 0.015s sys 0.012s I0918 20:39:58.476325 30507 tablet_bootstrap.cc:439] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac: No bootstrap required, opened a new log I0918 20:39:58.477660 30508 raft_consensus.cc:340] T f2533ad3d1db420c890b4ec86118331a P aa62fa80d8be490fbb5a2d10eef5235f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } I0918 20:39:58.478458 30508 raft_consensus.cc:366] T f2533ad3d1db420c890b4ec86118331a P aa62fa80d8be490fbb5a2d10eef5235f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:58.478693 30508 raft_consensus.cc:705] T f2533ad3d1db420c890b4ec86118331a P aa62fa80d8be490fbb5a2d10eef5235f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa62fa80d8be490fbb5a2d10eef5235f, State: Initialized, Role: FOLLOWER I0918 20:39:58.479168 30507 ts_tablet_manager.cc:1085] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac: Time spent bootstrapping tablet: real 0.077s user 0.008s sys 0.000s I0918 20:39:58.479753 30509 tablet_bootstrap.cc:439] T f2533ad3d1db420c890b4ec86118331a P 04a5db21380e475189ae3c20003c489f: No bootstrap required, opened a new log I0918 20:39:58.479321 30508 consensus_queue.cc:229] T f2533ad3d1db420c890b4ec86118331a P aa62fa80d8be490fbb5a2d10eef5235f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } I0918 20:39:58.480615 30509 ts_tablet_manager.cc:1085] T f2533ad3d1db420c890b4ec86118331a P 04a5db21380e475189ae3c20003c489f: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.008s I0918 20:39:58.485882 30509 raft_consensus.cc:340] T f2533ad3d1db420c890b4ec86118331a P 04a5db21380e475189ae3c20003c489f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } I0918 20:39:58.490952 30509 raft_consensus.cc:366] T f2533ad3d1db420c890b4ec86118331a P 04a5db21380e475189ae3c20003c489f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:58.491422 30509 raft_consensus.cc:705] T f2533ad3d1db420c890b4ec86118331a P 04a5db21380e475189ae3c20003c489f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a5db21380e475189ae3c20003c489f, State: Initialized, Role: FOLLOWER I0918 20:39:58.487896 30507 raft_consensus.cc:340] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } I0918 20:39:58.492331 30507 raft_consensus.cc:366] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:58.492635 30507 raft_consensus.cc:705] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f56d73728210458c871a73e1eaf928ac, State: Initialized, Role: FOLLOWER I0918 20:39:58.493667 30507 consensus_queue.cc:229] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } I0918 20:39:58.504130 30509 consensus_queue.cc:229] T f2533ad3d1db420c890b4ec86118331a P 04a5db21380e475189ae3c20003c489f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } I0918 20:39:58.533556 30508 ts_tablet_manager.cc:1113] T f2533ad3d1db420c890b4ec86118331a P aa62fa80d8be490fbb5a2d10eef5235f: Time spent starting tablet: real 0.057s user 0.019s sys 0.008s I0918 20:39:58.534380 30508 tablet_bootstrap.cc:439] T c95a78d505f64699b76403bb22206589 P aa62fa80d8be490fbb5a2d10eef5235f: Bootstrap starting. I0918 20:39:58.534921 30507 ts_tablet_manager.cc:1113] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac: Time spent starting tablet: real 0.055s user 0.011s sys 0.022s I0918 20:39:58.535971 30507 tablet_bootstrap.cc:439] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac: Bootstrap starting. I0918 20:39:58.537281 30508 tablet_bootstrap.cc:590] T c95a78d505f64699b76403bb22206589 P aa62fa80d8be490fbb5a2d10eef5235f: No blocks or log segments found. Creating new log. I0918 20:39:58.541676 30509 ts_tablet_manager.cc:1113] T f2533ad3d1db420c890b4ec86118331a P 04a5db21380e475189ae3c20003c489f: Time spent starting tablet: real 0.061s user 0.003s sys 0.015s I0918 20:39:58.542666 30509 tablet_bootstrap.cc:439] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f: Bootstrap starting. I0918 20:39:58.546048 30509 tablet_bootstrap.cc:590] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f: No blocks or log segments found. Creating new log. I0918 20:39:58.549861 30507 tablet_bootstrap.cc:590] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac: No blocks or log segments found. Creating new log. I0918 20:39:58.559020 30507 tablet_bootstrap.cc:439] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac: No bootstrap required, opened a new log I0918 20:39:58.559638 30507 ts_tablet_manager.cc:1085] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.004s I0918 20:39:58.562096 30507 raft_consensus.cc:340] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } I0918 20:39:58.564204 30507 raft_consensus.cc:366] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:58.564447 30507 raft_consensus.cc:705] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f56d73728210458c871a73e1eaf928ac, State: Initialized, Role: FOLLOWER I0918 20:39:58.566221 30507 consensus_queue.cc:229] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } I0918 20:39:58.572882 30514 raft_consensus.cc:472] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:58.579011 30507 ts_tablet_manager.cc:1113] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s I0918 20:39:58.580111 30507 tablet_bootstrap.cc:439] T 6557cb6da784488b900635b19d9df9c7 P f56d73728210458c871a73e1eaf928ac: Bootstrap starting. I0918 20:39:58.583489 30507 tablet_bootstrap.cc:590] T 6557cb6da784488b900635b19d9df9c7 P f56d73728210458c871a73e1eaf928ac: No blocks or log segments found. Creating new log. I0918 20:39:58.585476 30509 tablet_bootstrap.cc:439] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f: No bootstrap required, opened a new log I0918 20:39:58.585995 30509 ts_tablet_manager.cc:1085] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.000s I0918 20:39:58.586632 30514 raft_consensus.cc:494] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } I0918 20:39:58.588124 30508 tablet_bootstrap.cc:439] T c95a78d505f64699b76403bb22206589 P aa62fa80d8be490fbb5a2d10eef5235f: No bootstrap required, opened a new log I0918 20:39:58.588690 30508 ts_tablet_manager.cc:1085] T c95a78d505f64699b76403bb22206589 P aa62fa80d8be490fbb5a2d10eef5235f: Time spent bootstrapping tablet: real 0.054s user 0.007s sys 0.001s I0918 20:39:58.589370 30509 raft_consensus.cc:340] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } I0918 20:39:58.590203 30509 raft_consensus.cc:366] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:58.590621 30508 raft_consensus.cc:340] T c95a78d505f64699b76403bb22206589 P aa62fa80d8be490fbb5a2d10eef5235f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } I0918 20:39:58.591738 30508 raft_consensus.cc:366] T c95a78d505f64699b76403bb22206589 P aa62fa80d8be490fbb5a2d10eef5235f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:58.592005 30508 raft_consensus.cc:705] T c95a78d505f64699b76403bb22206589 P aa62fa80d8be490fbb5a2d10eef5235f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa62fa80d8be490fbb5a2d10eef5235f, State: Initialized, Role: FOLLOWER I0918 20:39:58.592759 30508 consensus_queue.cc:229] T c95a78d505f64699b76403bb22206589 P aa62fa80d8be490fbb5a2d10eef5235f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } I0918 20:39:58.601387 30509 raft_consensus.cc:705] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a5db21380e475189ae3c20003c489f, State: Initialized, Role: FOLLOWER I0918 20:39:58.602957 30509 consensus_queue.cc:229] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } I0918 20:39:58.603813 30508 ts_tablet_manager.cc:1113] T c95a78d505f64699b76403bb22206589 P aa62fa80d8be490fbb5a2d10eef5235f: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s I0918 20:39:58.609153 30514 leader_election.cc:296] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04a5db21380e475189ae3c20003c489f (127.18.222.2:38355), aa62fa80d8be490fbb5a2d10eef5235f (127.18.222.1:34719) I0918 20:39:58.612845 30508 tablet_bootstrap.cc:439] T 6557cb6da784488b900635b19d9df9c7 P aa62fa80d8be490fbb5a2d10eef5235f: Bootstrap starting. I0918 20:39:58.615252 30507 tablet_bootstrap.cc:439] T 6557cb6da784488b900635b19d9df9c7 P f56d73728210458c871a73e1eaf928ac: No bootstrap required, opened a new log I0918 20:39:58.615924 30507 ts_tablet_manager.cc:1085] T 6557cb6da784488b900635b19d9df9c7 P f56d73728210458c871a73e1eaf928ac: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.000s I0918 20:39:58.618096 30507 raft_consensus.cc:340] T 6557cb6da784488b900635b19d9df9c7 P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } I0918 20:39:58.622613 30508 tablet_bootstrap.cc:590] T 6557cb6da784488b900635b19d9df9c7 P aa62fa80d8be490fbb5a2d10eef5235f: No blocks or log segments found. Creating new log. I0918 20:39:58.622742 30507 raft_consensus.cc:366] T 6557cb6da784488b900635b19d9df9c7 P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:58.628424 30509 ts_tablet_manager.cc:1113] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s I0918 20:39:58.629384 30509 tablet_bootstrap.cc:439] T c95a78d505f64699b76403bb22206589 P 04a5db21380e475189ae3c20003c489f: Bootstrap starting. I0918 20:39:58.627614 30507 raft_consensus.cc:705] T 6557cb6da784488b900635b19d9df9c7 P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f56d73728210458c871a73e1eaf928ac, State: Initialized, Role: FOLLOWER I0918 20:39:58.632795 30509 tablet_bootstrap.cc:590] T c95a78d505f64699b76403bb22206589 P 04a5db21380e475189ae3c20003c489f: No blocks or log segments found. Creating new log. I0918 20:39:58.631583 30507 consensus_queue.cc:229] T 6557cb6da784488b900635b19d9df9c7 P f56d73728210458c871a73e1eaf928ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } I0918 20:39:58.641067 30507 ts_tablet_manager.cc:1113] T 6557cb6da784488b900635b19d9df9c7 P f56d73728210458c871a73e1eaf928ac: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s I0918 20:39:58.670882 30508 tablet_bootstrap.cc:439] T 6557cb6da784488b900635b19d9df9c7 P aa62fa80d8be490fbb5a2d10eef5235f: No bootstrap required, opened a new log I0918 20:39:58.671620 30508 ts_tablet_manager.cc:1085] T 6557cb6da784488b900635b19d9df9c7 P aa62fa80d8be490fbb5a2d10eef5235f: Time spent bootstrapping tablet: real 0.059s user 0.016s sys 0.018s I0918 20:39:58.674202 30508 raft_consensus.cc:340] T 6557cb6da784488b900635b19d9df9c7 P aa62fa80d8be490fbb5a2d10eef5235f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } I0918 20:39:58.674973 30508 raft_consensus.cc:366] T 6557cb6da784488b900635b19d9df9c7 P aa62fa80d8be490fbb5a2d10eef5235f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:58.675195 30508 raft_consensus.cc:705] T 6557cb6da784488b900635b19d9df9c7 P aa62fa80d8be490fbb5a2d10eef5235f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa62fa80d8be490fbb5a2d10eef5235f, State: Initialized, Role: FOLLOWER I0918 20:39:58.676126 30508 consensus_queue.cc:229] T 6557cb6da784488b900635b19d9df9c7 P aa62fa80d8be490fbb5a2d10eef5235f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } I0918 20:39:58.671671 30509 tablet_bootstrap.cc:439] T c95a78d505f64699b76403bb22206589 P 04a5db21380e475189ae3c20003c489f: No bootstrap required, opened a new log I0918 20:39:58.680207 30509 ts_tablet_manager.cc:1085] T c95a78d505f64699b76403bb22206589 P 04a5db21380e475189ae3c20003c489f: Time spent bootstrapping tablet: real 0.051s user 0.018s sys 0.014s I0918 20:39:58.683498 30509 raft_consensus.cc:340] T c95a78d505f64699b76403bb22206589 P 04a5db21380e475189ae3c20003c489f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } I0918 20:39:58.684140 30508 ts_tablet_manager.cc:1113] T 6557cb6da784488b900635b19d9df9c7 P aa62fa80d8be490fbb5a2d10eef5235f: Time spent starting tablet: real 0.012s user 0.002s sys 0.003s I0918 20:39:58.686668 30513 raft_consensus.cc:472] T c95a78d505f64699b76403bb22206589 P aa62fa80d8be490fbb5a2d10eef5235f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:58.687227 30513 raft_consensus.cc:494] T c95a78d505f64699b76403bb22206589 P aa62fa80d8be490fbb5a2d10eef5235f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } I0918 20:39:58.684696 30509 raft_consensus.cc:366] T c95a78d505f64699b76403bb22206589 P 04a5db21380e475189ae3c20003c489f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0918 20:39:58.689992 30513 leader_election.cc:296] T c95a78d505f64699b76403bb22206589 P aa62fa80d8be490fbb5a2d10eef5235f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f56d73728210458c871a73e1eaf928ac (127.18.222.3:42979), 04a5db21380e475189ae3c20003c489f (127.18.222.2:38355) I0918 20:39:58.689975 30509 raft_consensus.cc:705] T c95a78d505f64699b76403bb22206589 P 04a5db21380e475189ae3c20003c489f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a5db21380e475189ae3c20003c489f, State: Initialized, Role: FOLLOWER I0918 20:39:58.692613 30509 consensus_queue.cc:229] T c95a78d505f64699b76403bb22206589 P 04a5db21380e475189ae3c20003c489f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } I0918 20:39:58.699586 30509 ts_tablet_manager.cc:1113] T c95a78d505f64699b76403bb22206589 P 04a5db21380e475189ae3c20003c489f: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s I0918 20:39:58.712904 30384 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f2533ad3d1db420c890b4ec86118331a" candidate_uuid: "f56d73728210458c871a73e1eaf928ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a5db21380e475189ae3c20003c489f" is_pre_election: true I0918 20:39:58.713711 30384 raft_consensus.cc:2350] T f2533ad3d1db420c890b4ec86118331a P 04a5db21380e475189ae3c20003c489f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f56d73728210458c871a73e1eaf928ac in term 0. I0918 20:39:58.714802 30419 leader_election.cc:310] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a5db21380e475189ae3c20003c489f, f56d73728210458c871a73e1eaf928ac; no voters: I0918 20:39:58.715675 30514 raft_consensus.cc:2644] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:58.715981 30514 raft_consensus.cc:472] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:58.716231 30514 raft_consensus.cc:2890] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:58.722309 30514 raft_consensus.cc:494] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } I0918 20:39:58.725062 30384 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f2533ad3d1db420c890b4ec86118331a" candidate_uuid: "f56d73728210458c871a73e1eaf928ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a5db21380e475189ae3c20003c489f" I0918 20:39:58.725687 30384 raft_consensus.cc:2890] T f2533ad3d1db420c890b4ec86118331a P 04a5db21380e475189ae3c20003c489f [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:58.731060 30384 raft_consensus.cc:2350] T f2533ad3d1db420c890b4ec86118331a P 04a5db21380e475189ae3c20003c489f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f56d73728210458c871a73e1eaf928ac in term 1. I0918 20:39:58.732054 30419 leader_election.cc:310] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a5db21380e475189ae3c20003c489f, f56d73728210458c871a73e1eaf928ac; no voters: I0918 20:39:58.742441 30513 raft_consensus.cc:472] T f2533ad3d1db420c890b4ec86118331a P aa62fa80d8be490fbb5a2d10eef5235f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:58.743029 30513 raft_consensus.cc:494] T f2533ad3d1db420c890b4ec86118331a P aa62fa80d8be490fbb5a2d10eef5235f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } I0918 20:39:58.744779 30513 leader_election.cc:296] T f2533ad3d1db420c890b4ec86118331a P aa62fa80d8be490fbb5a2d10eef5235f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04a5db21380e475189ae3c20003c489f (127.18.222.2:38355), f56d73728210458c871a73e1eaf928ac (127.18.222.3:42979) I0918 20:39:58.745689 30313 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f2533ad3d1db420c890b4ec86118331a" candidate_uuid: "f56d73728210458c871a73e1eaf928ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" is_pre_election: true I0918 20:39:58.745738 30312 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f2533ad3d1db420c890b4ec86118331a" candidate_uuid: "f56d73728210458c871a73e1eaf928ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" I0918 20:39:58.746688 30313 raft_consensus.cc:2350] T f2533ad3d1db420c890b4ec86118331a P aa62fa80d8be490fbb5a2d10eef5235f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f56d73728210458c871a73e1eaf928ac in term 0. I0918 20:39:58.747316 30312 raft_consensus.cc:2325] T f2533ad3d1db420c890b4ec86118331a P aa62fa80d8be490fbb5a2d10eef5235f [term 0 FOLLOWER]: Leader election vote request: Denying vote to candidate f56d73728210458c871a73e1eaf928ac for term 1 because replica is already servicing an update from a current leader or another vote. I0918 20:39:58.738268 30514 leader_election.cc:296] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [CANDIDATE]: Term 1 election: Requested vote from peers 04a5db21380e475189ae3c20003c489f (127.18.222.2:38355), aa62fa80d8be490fbb5a2d10eef5235f (127.18.222.1:34719) I0918 20:39:58.749182 30514 raft_consensus.cc:472] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:58.749541 30515 raft_consensus.cc:472] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:58.750072 30515 raft_consensus.cc:494] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } I0918 20:39:58.754451 30515 leader_election.cc:296] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa62fa80d8be490fbb5a2d10eef5235f (127.18.222.1:34719), f56d73728210458c871a73e1eaf928ac (127.18.222.3:42979) I0918 20:39:58.753319 30525 raft_consensus.cc:2644] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:58.749567 30514 raft_consensus.cc:494] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } I0918 20:39:58.758795 30525 raft_consensus.cc:667] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [term 1 LEADER]: Becoming Leader. State: Replica: f56d73728210458c871a73e1eaf928ac, State: Running, Role: LEADER I0918 20:39:58.766674 30514 leader_election.cc:296] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa62fa80d8be490fbb5a2d10eef5235f (127.18.222.1:34719), 04a5db21380e475189ae3c20003c489f (127.18.222.2:38355) I0918 20:39:58.773486 30525 consensus_queue.cc:206] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } I0918 20:39:58.773999 30384 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c95a78d505f64699b76403bb22206589" candidate_uuid: "f56d73728210458c871a73e1eaf928ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a5db21380e475189ae3c20003c489f" is_pre_election: true I0918 20:39:58.781447 30384 raft_consensus.cc:2350] T c95a78d505f64699b76403bb22206589 P 04a5db21380e475189ae3c20003c489f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f56d73728210458c871a73e1eaf928ac in term 0. I0918 20:39:58.789016 30419 leader_election.cc:310] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a5db21380e475189ae3c20003c489f, f56d73728210458c871a73e1eaf928ac; no voters: I0918 20:39:58.805037 30526 raft_consensus.cc:2644] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:58.805362 30526 raft_consensus.cc:472] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:58.805653 30526 raft_consensus.cc:2890] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:58.812377 30526 raft_consensus.cc:494] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } I0918 20:39:58.815577 30384 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c95a78d505f64699b76403bb22206589" candidate_uuid: "f56d73728210458c871a73e1eaf928ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a5db21380e475189ae3c20003c489f" I0918 20:39:58.816268 30384 raft_consensus.cc:2890] T c95a78d505f64699b76403bb22206589 P 04a5db21380e475189ae3c20003c489f [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:58.819770 30526 leader_election.cc:296] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [CANDIDATE]: Term 1 election: Requested vote from peers aa62fa80d8be490fbb5a2d10eef5235f (127.18.222.1:34719), 04a5db21380e475189ae3c20003c489f (127.18.222.2:38355) I0918 20:39:58.823437 30384 raft_consensus.cc:2350] T c95a78d505f64699b76403bb22206589 P 04a5db21380e475189ae3c20003c489f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f56d73728210458c871a73e1eaf928ac in term 1. I0918 20:39:58.826439 30419 leader_election.cc:310] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a5db21380e475189ae3c20003c489f, f56d73728210458c871a73e1eaf928ac; no voters: I0918 20:39:58.827487 30526 raft_consensus.cc:2644] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:58.828352 30526 raft_consensus.cc:667] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [term 1 LEADER]: Becoming Leader. State: Replica: f56d73728210458c871a73e1eaf928ac, State: Running, Role: LEADER I0918 20:39:58.828588 30455 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c95a78d505f64699b76403bb22206589" candidate_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f56d73728210458c871a73e1eaf928ac" is_pre_election: true I0918 20:39:58.829056 30526 consensus_queue.cc:206] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } I0918 20:39:58.830240 30454 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f2533ad3d1db420c890b4ec86118331a" candidate_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f56d73728210458c871a73e1eaf928ac" is_pre_election: true I0918 20:39:58.874259 30384 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c95a78d505f64699b76403bb22206589" candidate_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a5db21380e475189ae3c20003c489f" is_pre_election: true I0918 20:39:58.875062 30384 raft_consensus.cc:2280] T c95a78d505f64699b76403bb22206589 P 04a5db21380e475189ae3c20003c489f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aa62fa80d8be490fbb5a2d10eef5235f in current term 1: Already voted for candidate f56d73728210458c871a73e1eaf928ac in this term. I0918 20:39:58.875190 30383 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f2533ad3d1db420c890b4ec86118331a" candidate_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a5db21380e475189ae3c20003c489f" is_pre_election: true I0918 20:39:58.877118 30383 raft_consensus.cc:2280] T f2533ad3d1db420c890b4ec86118331a P 04a5db21380e475189ae3c20003c489f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aa62fa80d8be490fbb5a2d10eef5235f in current term 1: Already voted for candidate f56d73728210458c871a73e1eaf928ac in this term. I0918 20:39:58.877420 30454 raft_consensus.cc:2311] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate aa62fa80d8be490fbb5a2d10eef5235f for term 1 because replica is either leader or believes a valid leader to be alive. I0918 20:39:58.879601 30269 leader_election.cc:310] T f2533ad3d1db420c890b4ec86118331a P aa62fa80d8be490fbb5a2d10eef5235f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa62fa80d8be490fbb5a2d10eef5235f; no voters: 04a5db21380e475189ae3c20003c489f, f56d73728210458c871a73e1eaf928ac I0918 20:39:58.880991 30513 raft_consensus.cc:2890] T f2533ad3d1db420c890b4ec86118331a P aa62fa80d8be490fbb5a2d10eef5235f [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:58.891402 30513 raft_consensus.cc:2592] T f2533ad3d1db420c890b4ec86118331a P aa62fa80d8be490fbb5a2d10eef5235f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0918 20:39:58.891515 30312 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6557cb6da784488b900635b19d9df9c7" candidate_uuid: "04a5db21380e475189ae3c20003c489f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" is_pre_election: true I0918 20:39:58.892966 30312 raft_consensus.cc:2350] T 6557cb6da784488b900635b19d9df9c7 P aa62fa80d8be490fbb5a2d10eef5235f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04a5db21380e475189ae3c20003c489f in term 0. I0918 20:39:58.893188 30455 raft_consensus.cc:2311] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate aa62fa80d8be490fbb5a2d10eef5235f for term 1 because replica is either leader or believes a valid leader to be alive. I0918 20:39:58.894351 30345 leader_election.cc:310] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a5db21380e475189ae3c20003c489f, aa62fa80d8be490fbb5a2d10eef5235f; no voters: I0918 20:39:58.895347 30515 raft_consensus.cc:2644] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [term 0 FOLLOWER]: Leader pre-election won for term 1 I0918 20:39:58.895639 30515 raft_consensus.cc:472] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0918 20:39:58.896742 30515 raft_consensus.cc:2890] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:58.900359 30269 leader_election.cc:310] T c95a78d505f64699b76403bb22206589 P aa62fa80d8be490fbb5a2d10eef5235f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa62fa80d8be490fbb5a2d10eef5235f; no voters: 04a5db21380e475189ae3c20003c489f, f56d73728210458c871a73e1eaf928ac I0918 20:39:58.903148 30515 raft_consensus.cc:494] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } I0918 20:39:58.905650 30312 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6557cb6da784488b900635b19d9df9c7" candidate_uuid: "04a5db21380e475189ae3c20003c489f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" I0918 20:39:58.906265 30312 raft_consensus.cc:2890] T 6557cb6da784488b900635b19d9df9c7 P aa62fa80d8be490fbb5a2d10eef5235f [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:58.907961 30311 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c95a78d505f64699b76403bb22206589" candidate_uuid: "f56d73728210458c871a73e1eaf928ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" I0918 20:39:58.908651 30311 raft_consensus.cc:2890] T c95a78d505f64699b76403bb22206589 P aa62fa80d8be490fbb5a2d10eef5235f [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:58.911016 30313 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c95a78d505f64699b76403bb22206589" candidate_uuid: "f56d73728210458c871a73e1eaf928ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" is_pre_election: true I0918 20:39:58.913823 30312 raft_consensus.cc:2350] T 6557cb6da784488b900635b19d9df9c7 P aa62fa80d8be490fbb5a2d10eef5235f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04a5db21380e475189ae3c20003c489f in term 1. I0918 20:39:58.916206 30345 leader_election.cc:310] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a5db21380e475189ae3c20003c489f, aa62fa80d8be490fbb5a2d10eef5235f; no voters: I0918 20:39:58.919214 30311 raft_consensus.cc:2350] T c95a78d505f64699b76403bb22206589 P aa62fa80d8be490fbb5a2d10eef5235f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f56d73728210458c871a73e1eaf928ac in term 1. I0918 20:39:58.921175 30513 raft_consensus.cc:2592] T c95a78d505f64699b76403bb22206589 P aa62fa80d8be490fbb5a2d10eef5235f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0918 20:39:58.922446 30313 raft_consensus.cc:2325] T c95a78d505f64699b76403bb22206589 P aa62fa80d8be490fbb5a2d10eef5235f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f56d73728210458c871a73e1eaf928ac for term 1 because replica is already servicing an update from a current leader or another vote. I0918 20:39:58.912894 30515 leader_election.cc:296] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [CANDIDATE]: Term 1 election: Requested vote from peers aa62fa80d8be490fbb5a2d10eef5235f (127.18.222.1:34719), f56d73728210458c871a73e1eaf928ac (127.18.222.3:42979) I0918 20:39:58.924113 30515 raft_consensus.cc:2644] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [term 1 FOLLOWER]: Leader election won for term 1 I0918 20:39:58.924510 30515 raft_consensus.cc:667] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [term 1 LEADER]: Becoming Leader. State: Replica: 04a5db21380e475189ae3c20003c489f, State: Running, Role: LEADER I0918 20:39:58.925104 30515 consensus_queue.cc:206] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } I0918 20:39:58.929994 30526 raft_consensus.cc:472] T 6557cb6da784488b900635b19d9df9c7 P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0918 20:39:58.930722 30526 raft_consensus.cc:494] T 6557cb6da784488b900635b19d9df9c7 P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } } I0918 20:39:58.931895 30455 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6557cb6da784488b900635b19d9df9c7" candidate_uuid: "04a5db21380e475189ae3c20003c489f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f56d73728210458c871a73e1eaf928ac" is_pre_election: true I0918 20:39:58.932683 30455 raft_consensus.cc:2350] T 6557cb6da784488b900635b19d9df9c7 P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04a5db21380e475189ae3c20003c489f in term 0. I0918 20:39:58.934406 30455 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6557cb6da784488b900635b19d9df9c7" candidate_uuid: "04a5db21380e475189ae3c20003c489f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f56d73728210458c871a73e1eaf928ac" I0918 20:39:58.934934 30455 raft_consensus.cc:2890] T 6557cb6da784488b900635b19d9df9c7 P f56d73728210458c871a73e1eaf928ac [term 0 FOLLOWER]: Advancing to term 1 I0918 20:39:58.905201 30155 catalog_manager.cc:4131] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac reported cstate change: term changed from 0 to 1, leader changed from to f56d73728210458c871a73e1eaf928ac (127.18.222.3). New cstate: current_term: 1 leader_uuid: "f56d73728210458c871a73e1eaf928ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } health_report { overall_health: HEALTHY } } } I0918 20:39:58.940346 30383 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6557cb6da784488b900635b19d9df9c7" candidate_uuid: "f56d73728210458c871a73e1eaf928ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a5db21380e475189ae3c20003c489f" is_pre_election: true I0918 20:39:58.942476 30313 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6557cb6da784488b900635b19d9df9c7" candidate_uuid: "f56d73728210458c871a73e1eaf928ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" is_pre_election: true I0918 20:39:58.943378 30313 raft_consensus.cc:2280] T 6557cb6da784488b900635b19d9df9c7 P aa62fa80d8be490fbb5a2d10eef5235f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f56d73728210458c871a73e1eaf928ac in current term 1: Already voted for candidate 04a5db21380e475189ae3c20003c489f in this term. I0918 20:39:58.944391 30526 leader_election.cc:296] T 6557cb6da784488b900635b19d9df9c7 P f56d73728210458c871a73e1eaf928ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa62fa80d8be490fbb5a2d10eef5235f (127.18.222.1:34719), 04a5db21380e475189ae3c20003c489f (127.18.222.2:38355) I0918 20:39:58.947238 30383 raft_consensus.cc:2311] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate f56d73728210458c871a73e1eaf928ac for term 1 because replica is either leader or believes a valid leader to be alive. I0918 20:39:58.950112 30455 raft_consensus.cc:2350] T 6557cb6da784488b900635b19d9df9c7 P f56d73728210458c871a73e1eaf928ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04a5db21380e475189ae3c20003c489f in term 1. I0918 20:39:58.950386 30416 leader_election.cc:310] T 6557cb6da784488b900635b19d9df9c7 P f56d73728210458c871a73e1eaf928ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f56d73728210458c871a73e1eaf928ac; no voters: 04a5db21380e475189ae3c20003c489f, aa62fa80d8be490fbb5a2d10eef5235f I0918 20:39:58.952560 30526 raft_consensus.cc:2592] T 6557cb6da784488b900635b19d9df9c7 P f56d73728210458c871a73e1eaf928ac [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0918 20:39:58.960655 30158 catalog_manager.cc:4131] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f reported cstate change: term changed from 0 to 1, leader changed from to 04a5db21380e475189ae3c20003c489f (127.18.222.2). New cstate: current_term: 1 leader_uuid: "04a5db21380e475189ae3c20003c489f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } health_report { overall_health: UNKNOWN } } } I0918 20:39:58.998955 30156 catalog_manager.cc:4131] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac reported cstate change: term changed from 0 to 1, leader changed from to f56d73728210458c871a73e1eaf928ac (127.18.222.3). New cstate: current_term: 1 leader_uuid: "f56d73728210458c871a73e1eaf928ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 } health_report { overall_health: UNKNOWN } } } W0918 20:39:59.072125 30140 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.222.62:43789, user_credentials={real_user=slave}} blocked reactor thread for 48520.6us I0918 20:39:59.235874 30526 consensus_queue.cc:984] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:59.261550 30526 consensus_queue.cc:984] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:59.296896 30526 consensus_queue.cc:984] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s I0918 20:39:59.344540 30526 consensus_queue.cc:984] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "04a5db21380e475189ae3c20003c489f" member_type: VOTER last_known_addr { host: "127.18.222.2" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:59.416169 30535 consensus_queue.cc:984] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa62fa80d8be490fbb5a2d10eef5235f" member_type: VOTER last_known_addr { host: "127.18.222.1" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:59.439661 30515 consensus_queue.cc:984] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f56d73728210458c871a73e1eaf928ac" member_type: VOTER last_known_addr { host: "127.18.222.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0918 20:39:59.889459 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:39:59.971263 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:39:59.971747 19320 tablet_replica.cc:273] T 6557cb6da784488b900635b19d9df9c7 P aa62fa80d8be490fbb5a2d10eef5235f: stopping tablet replica I0918 20:39:59.972266 19320 raft_consensus.cc:2147] T 6557cb6da784488b900635b19d9df9c7 P aa62fa80d8be490fbb5a2d10eef5235f [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:59.972637 19320 raft_consensus.cc:2174] T 6557cb6da784488b900635b19d9df9c7 P aa62fa80d8be490fbb5a2d10eef5235f [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:59.975426 19320 tablet_replica.cc:273] T c95a78d505f64699b76403bb22206589 P aa62fa80d8be490fbb5a2d10eef5235f: stopping tablet replica I0918 20:39:59.975860 19320 raft_consensus.cc:2147] T c95a78d505f64699b76403bb22206589 P aa62fa80d8be490fbb5a2d10eef5235f [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:59.976174 19320 raft_consensus.cc:2174] T c95a78d505f64699b76403bb22206589 P aa62fa80d8be490fbb5a2d10eef5235f [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:39:59.981983 19320 tablet_replica.cc:273] T f2533ad3d1db420c890b4ec86118331a P aa62fa80d8be490fbb5a2d10eef5235f: stopping tablet replica I0918 20:39:59.982375 19320 raft_consensus.cc:2147] T f2533ad3d1db420c890b4ec86118331a P aa62fa80d8be490fbb5a2d10eef5235f [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:39:59.982762 19320 raft_consensus.cc:2174] T f2533ad3d1db420c890b4ec86118331a P aa62fa80d8be490fbb5a2d10eef5235f [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:40:00.016183 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:40:00.016391 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:40:00.016716 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:40:00.031426 19320 tablet_server.cc:155] TabletServer shutting down... W0918 20:40:00.037884 30345 consensus_peers.cc:458] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f -> Peer aa62fa80d8be490fbb5a2d10eef5235f (127.18.222.1:34719): Couldn't send request to peer aa62fa80d8be490fbb5a2d10eef5235f. Status: Network error: Client connection negotiation failed: client connection to 127.18.222.1:34719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0918 20:40:00.079113 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:40:00.079761 19320 tablet_replica.cc:273] T c95a78d505f64699b76403bb22206589 P 04a5db21380e475189ae3c20003c489f: stopping tablet replica I0918 20:40:00.080250 19320 raft_consensus.cc:2147] T c95a78d505f64699b76403bb22206589 P 04a5db21380e475189ae3c20003c489f [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:40:00.080571 19320 raft_consensus.cc:2174] T c95a78d505f64699b76403bb22206589 P 04a5db21380e475189ae3c20003c489f [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:40:00.083462 19320 tablet_replica.cc:273] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f: stopping tablet replica I0918 20:40:00.083922 19320 raft_consensus.cc:2147] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [term 1 LEADER]: Raft consensus shutting down. I0918 20:40:00.084720 19320 raft_consensus.cc:2174] T 6557cb6da784488b900635b19d9df9c7 P 04a5db21380e475189ae3c20003c489f [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:40:00.087551 19320 tablet_replica.cc:273] T f2533ad3d1db420c890b4ec86118331a P 04a5db21380e475189ae3c20003c489f: stopping tablet replica I0918 20:40:00.088094 19320 raft_consensus.cc:2147] T f2533ad3d1db420c890b4ec86118331a P 04a5db21380e475189ae3c20003c489f [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:40:00.088457 19320 raft_consensus.cc:2174] T f2533ad3d1db420c890b4ec86118331a P 04a5db21380e475189ae3c20003c489f [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:40:00.114300 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:40:00.114527 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:40:00.114887 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:40:00.127058 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:40:00.156615 19320 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0918 20:40:00.156942 19320 tablet_replica.cc:273] T 6557cb6da784488b900635b19d9df9c7 P f56d73728210458c871a73e1eaf928ac: stopping tablet replica I0918 20:40:00.157320 19320 raft_consensus.cc:2147] T 6557cb6da784488b900635b19d9df9c7 P f56d73728210458c871a73e1eaf928ac [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:40:00.157616 19320 raft_consensus.cc:2174] T 6557cb6da784488b900635b19d9df9c7 P f56d73728210458c871a73e1eaf928ac [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:40:00.160307 19320 tablet_replica.cc:273] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac: stopping tablet replica I0918 20:40:00.160681 19320 raft_consensus.cc:2147] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [term 1 LEADER]: Raft consensus shutting down. I0918 20:40:00.161365 19320 raft_consensus.cc:2174] T c95a78d505f64699b76403bb22206589 P f56d73728210458c871a73e1eaf928ac [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:40:00.164321 19320 tablet_replica.cc:273] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac: stopping tablet replica I0918 20:40:00.164697 19320 raft_consensus.cc:2147] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [term 1 LEADER]: Raft consensus shutting down. I0918 20:40:00.165360 19320 raft_consensus.cc:2174] T f2533ad3d1db420c890b4ec86118331a P f56d73728210458c871a73e1eaf928ac [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:40:00.181111 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:40:00.181330 19320 tablet_server.cc:155] TabletServer shutting down... I0918 20:40:00.181692 19320 tablet_server.cc:169] TabletServer shutdown complete. I0918 20:40:00.191879 19320 master.cc:247] Master@127.18.222.62:43789 shutting down... I0918 20:40:00.211654 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:40:00.212113 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:40:00.212374 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P 09058351a47546a7a1d3065bb052c0a7: stopping tablet replica I0918 20:40:00.230849 19320 master.cc:258] Master@127.18.222.62:43789 shutdown complete. I0918 20:40:00.240603 19320 master.cc:247] Master@127.18.222.61:45529 shutting down... I0918 20:40:00.259258 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [term 1 LEADER]: Raft consensus shutting down. I0918 20:40:00.260150 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:40:00.260417 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P de19abd28eca4f258b52bf7ba47a80a6: stopping tablet replica I0918 20:40:00.279955 19320 master.cc:258] Master@127.18.222.61:45529 shutdown complete. I0918 20:40:00.291456 19320 master.cc:247] Master@127.18.222.60:34087 shutting down... I0918 20:40:00.310530 19320 raft_consensus.cc:2147] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160 [term 1 FOLLOWER]: Raft consensus shutting down. I0918 20:40:00.310878 19320 raft_consensus.cc:2174] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160 [term 1 FOLLOWER]: Raft consensus is shut down! I0918 20:40:00.311084 19320 tablet_replica.cc:273] T 00000000000000000000000000000000 P e3d8bb9e6e304aaea1273fc041c2f160: stopping tablet replica I0918 20:40:00.329023 19320 master.cc:258] Master@127.18.222.60:34087 shutdown complete. [ OK ] RemoteKsckTest.TestTabletFilters (3379 ms) [----------] 21 tests from RemoteKsckTest (110363 ms total) [----------] Global test environment tear-down [==========] 21 tests from 1 test case ran. (110363 ms total) [ PASSED ] 21 tests. ThreadSanitizer: reported 4 warnings