Note: This is test shard 1 of 8. [==========] Running 2 tests from 2 test cases. [----------] Global test environment set-up. [----------] 1 test from MaintenanceModeRF3ITest [ RUN ] MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate 2020-02-20T18:44:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG) 2020-02-20T18:44:24Z Disabled control of system clock WARNING: Logging before InitGoogleLogging() is written to STDERR I0220 18:44:24.281708 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-master /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.23.135.190:45127 --webserver_interface=127.23.135.190 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:41009 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:44:24.334549 24102 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:44:24.334699 24102 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:44:24.335784 24102 flags.cc:403] Enabled experimental flag: --ipki_ca_key_size=1024 W0220 18:44:24.335808 24102 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:44:24.335815 24102 flags.cc:403] Enabled experimental flag: --tsk_num_rsa_bits=512 W0220 18:44:24.335826 24102 flags.cc:403] Enabled experimental flag: --rpc_reuseport=true I0220 18:44:24.337443 24102 master_runner.cc:90] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:41009 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.23.135.190:45127 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/data/info.pb --webserver_interface=127.23.135.190 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.24102 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-master --enable_minidumps=false --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:44:24.337935 24102 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:44:24.338610 24102 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:44:24.343295 24102 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:44:24.344230 24102 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:44:24.345363 24102 hybrid_clock.cc:552] HybridClock initialized: now 1582224264345316 us; error 54 us; skew 500 ppm I0220 18:44:24.345888 24102 fs_manager.cc:259] Metadata directory not provided I0220 18:44:24.345922 24102 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:44:24.346129 24102 server_base.cc:502] Could not load existing FS layout: Not found: could not find a healthy instance file I0220 18:44:24.346148 24102 server_base.cc:503] Attempting to create new FS layout instead I0220 18:44:24.347856 24102 fs_manager.cc:613] Generated new instance metadata in path /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/data/instance: uuid: "1b8314706b1247e7a519c16d52957797" format_stamp: "Formatted at 2020-02-20 18:44:24 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:24.348138 24102 fs_manager.cc:613] Generated new instance metadata in path /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/wal/instance: uuid: "1b8314706b1247e7a519c16d52957797" format_stamp: "Formatted at 2020-02-20 18:44:24 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:24.349632 24102 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/data/data/block_manager_instance: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/data/data/block_manager_instance: No such file or directory (error 2) I0220 18:44:24.354027 24102 fs_manager.cc:514] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s I0220 18:44:24.357447 24102 fs_manager.cc:430] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0220 18:44:24.357522 24102 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/wal uuid: "1b8314706b1247e7a519c16d52957797" format_stamp: "Formatted at 2020-02-20 18:44:24 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:24.357609 24102 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:44:24.381023 24102 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:44:24.385699 24102 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.190:45127 I0220 18:44:24.386438 24102 webserver.cc:382] Webserver started at http://127.23.135.190:38291/ using document root and password file I0220 18:44:24.386914 24102 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/data/info.pb I0220 18:44:24.389300 24163 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:24.393469 24163 tablet_bootstrap.cc:441] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797: Bootstrap starting. I0220 18:44:24.395117 24163 tablet_bootstrap.cc:597] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797: No blocks or log segments found. Creating new log. I0220 18:44:24.395735 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-master as pid 24102 I0220 18:44:24.395821 24163 log.cc:828] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797: Log is configured to *not* fsync() on all Append() calls I0220 18:44:24.396250 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.23.135.129:0 --local_ip_for_outbound_sockets=127.23.135.129 --webserver_interface=127.23.135.129 --webserver_port=0 --tserver_master_addrs=127.23.135.190:45127 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:41009 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --follower_unavailable_considered_failed_sec=2 --enable_log_gc=false with env {} I0220 18:44:24.397156 24163 tablet_bootstrap.cc:441] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797: No bootstrap required, opened a new log I0220 18:44:24.399403 24163 raft_consensus.cc:346] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8314706b1247e7a519c16d52957797" member_type: VOTER } I0220 18:44:24.399462 24163 raft_consensus.cc:372] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:24.399511 24163 raft_consensus.cc:726] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b8314706b1247e7a519c16d52957797, State: Initialized, Role: FOLLOWER I0220 18:44:24.404016 24163 consensus_queue.cc:252] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8314706b1247e7a519c16d52957797" member_type: VOTER } I0220 18:44:24.404083 24163 raft_consensus.cc:386] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0220 18:44:24.404120 24163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0220 18:44:24.404191 24163 raft_consensus.cc:2983] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:24.404848 24163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8314706b1247e7a519c16d52957797" member_type: VOTER } I0220 18:44:24.405237 24163 leader_election.cc:307] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b8314706b1247e7a519c16d52957797; no voters: I0220 18:44:24.405530 24163 leader_election.cc:293] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [CANDIDATE]: Term 1 election: Requested vote from peers I0220 18:44:24.405656 24166 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [term 1 FOLLOWER]: Leader election won for term 1 I0220 18:44:24.405856 24166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [term 1 LEADER]: Becoming Leader. State: Replica: 1b8314706b1247e7a519c16d52957797, State: Running, Role: LEADER I0220 18:44:24.406177 24166 consensus_queue.cc:229] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8314706b1247e7a519c16d52957797" member_type: VOTER } I0220 18:44:24.407505 24166 sys_catalog.cc:365] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b8314706b1247e7a519c16d52957797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8314706b1247e7a519c16d52957797" member_type: VOTER } } I0220 18:44:24.407544 24166 sys_catalog.cc:368] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [sys.catalog]: This master's current role is: LEADER I0220 18:44:24.407774 24168 sys_catalog.cc:365] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b8314706b1247e7a519c16d52957797. Latest consensus state: current_term: 1 leader_uuid: "1b8314706b1247e7a519c16d52957797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8314706b1247e7a519c16d52957797" member_type: VOTER } } I0220 18:44:24.407797 24163 sys_catalog.cc:365] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "1b8314706b1247e7a519c16d52957797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8314706b1247e7a519c16d52957797" member_type: VOTER } } I0220 18:44:24.407831 24163 sys_catalog.cc:368] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [sys.catalog]: This master's current role is: LEADER I0220 18:44:24.407802 24168 sys_catalog.cc:368] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [sys.catalog]: This master's current role is: LEADER I0220 18:44:24.408478 24163 sys_catalog.cc:466] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [sys.catalog]: configured and running, proceeding with master startup. I0220 18:44:24.409365 24170 catalog_manager.cc:1107] Loading table and tablet metadata into memory... I0220 18:44:24.411597 24170 catalog_manager.cc:1118] Initializing Kudu internal certificate authority... I0220 18:44:24.431425 24170 catalog_manager.cc:1002] Generated new certificate authority record I0220 18:44:24.432369 24170 catalog_manager.cc:1127] Loading token signing keys... I0220 18:44:24.438365 24170 catalog_manager.cc:4509] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797: Generated new TSK 0 I0220 18:44:24.438732 24170 catalog_manager.cc:1137] Initializing in-progress tserver states... WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:44:24.491683 24165 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:44:24.491859 24165 flags.cc:403] Enabled unsafe flag: --enable_log_gc=false W0220 18:44:24.491890 24165 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:44:24.492810 24165 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:44:24.492866 24165 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.129 I0220 18:44:24.494272 24165 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:41009 --ntp_initial_sync_wait_secs=10 --time_source=builtin --follower_unavailable_considered_failed_sec=2 --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.129:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/data/info.pb --webserver_interface=127.23.135.129 --webserver_port=0 --enable_log_gc=false --tserver_master_addrs=127.23.135.190:45127 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.24165 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.129 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:44:24.494805 24165 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:44:24.495484 24165 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:44:24.508796 24165 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:44:24.509752 24165 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:44:24.517115 24165 hybrid_clock.cc:552] HybridClock initialized: now 1582224264517070 us; error 44 us; skew 500 ppm I0220 18:44:24.517827 24165 fs_manager.cc:259] Metadata directory not provided I0220 18:44:24.517874 24165 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:44:24.518085 24165 server_base.cc:502] Could not load existing FS layout: Not found: could not find a healthy instance file I0220 18:44:24.518108 24165 server_base.cc:503] Attempting to create new FS layout instead I0220 18:44:24.519980 24165 fs_manager.cc:613] Generated new instance metadata in path /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/data/instance: uuid: "493be4abe83d46c1829e3d5cfc68a9de" format_stamp: "Formatted at 2020-02-20 18:44:24 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:24.520274 24165 fs_manager.cc:613] Generated new instance metadata in path /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/wal/instance: uuid: "493be4abe83d46c1829e3d5cfc68a9de" format_stamp: "Formatted at 2020-02-20 18:44:24 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:24.522358 24165 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/data/data/block_manager_instance: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/data/data/block_manager_instance: No such file or directory (error 2) I0220 18:44:24.528390 24165 fs_manager.cc:514] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s I0220 18:44:24.532796 24165 fs_manager.cc:430] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0220 18:44:24.532930 24165 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/wal uuid: "493be4abe83d46c1829e3d5cfc68a9de" format_stamp: "Formatted at 2020-02-20 18:44:24 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:24.533025 24165 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:44:24.550750 24165 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:44:24.551893 24165 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0220 18:44:24.551930 24165 ts_tablet_manager.cc:415] Registered 0 tablets I0220 18:44:24.559181 24165 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.129:35559 I0220 18:44:24.560770 24165 webserver.cc:382] Webserver started at http://127.23.135.129:45061/ using document root and password file I0220 18:44:24.561336 24165 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/data/info.pb I0220 18:44:24.563408 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 24165 I0220 18:44:24.563925 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.23.135.130:0 --local_ip_for_outbound_sockets=127.23.135.130 --webserver_interface=127.23.135.130 --webserver_port=0 --tserver_master_addrs=127.23.135.190:45127 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:41009 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --follower_unavailable_considered_failed_sec=2 --enable_log_gc=false with env {} I0220 18:44:24.581792 24295 heartbeater.cc:322] Connected to a master server at 127.23.135.190:45127 I0220 18:44:24.581928 24295 heartbeater.cc:406] Registering TS with master... I0220 18:44:24.582289 24295 heartbeater.cc:455] Master 127.23.135.190:45127 requested a full tablet report, sending... I0220 18:44:24.583493 24130 ts_manager.cc:192] Registered new tserver with Master: 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559) I0220 18:44:24.585180 24130 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.129:41961 WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:44:24.636041 24297 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:44:24.636175 24297 flags.cc:403] Enabled unsafe flag: --enable_log_gc=false W0220 18:44:24.636186 24297 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:44:24.637034 24297 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:44:24.637080 24297 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.130 I0220 18:44:24.638351 24297 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:41009 --ntp_initial_sync_wait_secs=10 --time_source=builtin --follower_unavailable_considered_failed_sec=2 --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-1/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.130:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-1/data/info.pb --webserver_interface=127.23.135.130 --webserver_port=0 --enable_log_gc=false --tserver_master_addrs=127.23.135.190:45127 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.24297 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.130 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:44:24.638792 24297 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:44:24.639549 24297 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:44:24.644568 24297 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:44:24.645315 24297 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:44:24.646452 24297 hybrid_clock.cc:552] HybridClock initialized: now 1582224264646433 us; error 24 us; skew 500 ppm I0220 18:44:24.646996 24297 fs_manager.cc:259] Metadata directory not provided I0220 18:44:24.647032 24297 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:44:24.647248 24297 server_base.cc:502] Could not load existing FS layout: Not found: could not find a healthy instance file I0220 18:44:24.647270 24297 server_base.cc:503] Attempting to create new FS layout instead I0220 18:44:24.648816 24297 fs_manager.cc:613] Generated new instance metadata in path /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-1/data/instance: uuid: "c8a15b6a57804d05ada89b15de898a87" format_stamp: "Formatted at 2020-02-20 18:44:24 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:24.649083 24297 fs_manager.cc:613] Generated new instance metadata in path /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-1/wal/instance: uuid: "c8a15b6a57804d05ada89b15de898a87" format_stamp: "Formatted at 2020-02-20 18:44:24 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:24.650539 24297 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-1/data/data/block_manager_instance: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-1/data/data/block_manager_instance: No such file or directory (error 2) I0220 18:44:24.654721 24297 fs_manager.cc:514] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s I0220 18:44:24.658057 24297 fs_manager.cc:430] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0220 18:44:24.658131 24297 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-1/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-1/wal uuid: "c8a15b6a57804d05ada89b15de898a87" format_stamp: "Formatted at 2020-02-20 18:44:24 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:24.658234 24297 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-1/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-1/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:44:24.674063 24297 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:44:24.674880 24297 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0220 18:44:24.674914 24297 ts_tablet_manager.cc:415] Registered 0 tablets I0220 18:44:24.681279 24297 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.130:40621 I0220 18:44:24.681964 24297 webserver.cc:382] Webserver started at http://127.23.135.130:42851/ using document root and password file I0220 18:44:24.682394 24297 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-1/data/info.pb I0220 18:44:24.689738 24420 heartbeater.cc:322] Connected to a master server at 127.23.135.190:45127 I0220 18:44:24.689865 24420 heartbeater.cc:406] Registering TS with master... I0220 18:44:24.690186 24420 heartbeater.cc:455] Master 127.23.135.190:45127 requested a full tablet report, sending... I0220 18:44:24.690827 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 24297 I0220 18:44:24.690949 24130 ts_manager.cc:192] Registered new tserver with Master: c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621) I0220 18:44:24.691251 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.23.135.131:0 --local_ip_for_outbound_sockets=127.23.135.131 --webserver_interface=127.23.135.131 --webserver_port=0 --tserver_master_addrs=127.23.135.190:45127 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:41009 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --follower_unavailable_considered_failed_sec=2 --enable_log_gc=false with env {} I0220 18:44:24.691660 24130 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.130:34775 WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:44:24.735440 24423 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:44:24.735592 24423 flags.cc:403] Enabled unsafe flag: --enable_log_gc=false W0220 18:44:24.735607 24423 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:44:24.736548 24423 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:44:24.736632 24423 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.131 I0220 18:44:24.737942 24423 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:41009 --ntp_initial_sync_wait_secs=10 --time_source=builtin --follower_unavailable_considered_failed_sec=2 --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-2/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.131:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-2/data/info.pb --webserver_interface=127.23.135.131 --webserver_port=0 --enable_log_gc=false --tserver_master_addrs=127.23.135.190:45127 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.24423 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.131 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:44:24.738409 24423 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:44:24.739025 24423 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:44:24.743824 24423 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:44:24.744568 24423 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:44:24.745692 24423 hybrid_clock.cc:552] HybridClock initialized: now 1582224264745681 us; error 33 us; skew 500 ppm I0220 18:44:24.746192 24423 fs_manager.cc:259] Metadata directory not provided I0220 18:44:24.746224 24423 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:44:24.746387 24423 server_base.cc:502] Could not load existing FS layout: Not found: could not find a healthy instance file I0220 18:44:24.746404 24423 server_base.cc:503] Attempting to create new FS layout instead I0220 18:44:24.747845 24423 fs_manager.cc:613] Generated new instance metadata in path /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-2/data/instance: uuid: "bccadedc7a11414384f1d4875bf49aaa" format_stamp: "Formatted at 2020-02-20 18:44:24 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:24.748085 24423 fs_manager.cc:613] Generated new instance metadata in path /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-2/wal/instance: uuid: "bccadedc7a11414384f1d4875bf49aaa" format_stamp: "Formatted at 2020-02-20 18:44:24 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:24.749459 24423 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-2/data/data/block_manager_instance: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-2/data/data/block_manager_instance: No such file or directory (error 2) I0220 18:44:24.753494 24423 fs_manager.cc:514] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s I0220 18:44:24.756582 24423 fs_manager.cc:430] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0220 18:44:24.756659 24423 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-2/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-2/wal uuid: "bccadedc7a11414384f1d4875bf49aaa" format_stamp: "Formatted at 2020-02-20 18:44:24 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:24.756738 24423 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-2/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-2/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:44:24.778389 24423 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:44:24.779197 24423 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0220 18:44:24.779225 24423 ts_tablet_manager.cc:415] Registered 0 tablets I0220 18:44:24.784953 24423 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.131:46803 I0220 18:44:24.785665 24423 webserver.cc:382] Webserver started at http://127.23.135.131:37159/ using document root and password file I0220 18:44:24.786090 24423 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-2/data/info.pb I0220 18:44:24.793444 24544 heartbeater.cc:322] Connected to a master server at 127.23.135.190:45127 I0220 18:44:24.793565 24544 heartbeater.cc:406] Registering TS with master... I0220 18:44:24.793862 24544 heartbeater.cc:455] Master 127.23.135.190:45127 requested a full tablet report, sending... I0220 18:44:24.794584 24130 ts_manager.cc:192] Registered new tserver with Master: bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:24.795217 24130 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.131:55549 I0220 18:44:24.795665 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 24423 I0220 18:44:24.803661 24094 external_mini_cluster.cc:738] 3 TS(s) registered with all masters I0220 18:44:24.813557 24094 test_util.cc:228] Using random seed: 1262640114 I0220 18:44:24.823688 24130 catalog_manager.cc:1472] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40414: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\000UUU\025\004\001\000\252\252\252*\004\001\000\377\377\377?\004\001\000TUUU\004\001\000\251\252\252j" indirect_data: "" } partition_schema { range_schema { columns { name: "key" } } } W0220 18:44:24.824707 24130 catalog_manager.cc:1637] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0220 18:44:24.840596 24352 tablet_service.cc:1067] Processing CreateTablet for tablet c0ca382ac98b46948b2b18c2d691a364 (table=test-workload [id=73578aeb8f9e4f9597b79f5737a60111]), partition=RANGE (key) PARTITION VALUES >= 1789569705 I0220 18:44:24.841136 24352 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet c0ca382ac98b46948b2b18c2d691a364. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:24.841375 24481 tablet_service.cc:1067] Processing CreateTablet for tablet 116ca55e12164875ad0a87183fe531bc (table=test-workload [id=73578aeb8f9e4f9597b79f5737a60111]), partition=RANGE (key) PARTITION VALUES < 357913941 I0220 18:44:24.841377 24219 tablet_service.cc:1067] Processing CreateTablet for tablet 116ca55e12164875ad0a87183fe531bc (table=test-workload [id=73578aeb8f9e4f9597b79f5737a60111]), partition=RANGE (key) PARTITION VALUES < 357913941 I0220 18:44:24.841526 24478 tablet_service.cc:1067] Processing CreateTablet for tablet c5b2eb9c654f49df828ad8a096c62921 (table=test-workload [id=73578aeb8f9e4f9597b79f5737a60111]), partition=RANGE (key) PARTITION 1073741823 <= VALUES < 1431655764 I0220 18:44:24.841377 24476 tablet_service.cc:1067] Processing CreateTablet for tablet c0ca382ac98b46948b2b18c2d691a364 (table=test-workload [id=73578aeb8f9e4f9597b79f5737a60111]), partition=RANGE (key) PARTITION VALUES >= 1789569705 I0220 18:44:24.841806 24478 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet c5b2eb9c654f49df828ad8a096c62921. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:24.841807 24219 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 116ca55e12164875ad0a87183fe531bc. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:24.842043 24481 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 116ca55e12164875ad0a87183fe531bc. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:24.842698 24480 tablet_service.cc:1067] Processing CreateTablet for tablet a7a069dabd5b4df28954b6833f23ab31 (table=test-workload [id=73578aeb8f9e4f9597b79f5737a60111]), partition=RANGE (key) PARTITION 357913941 <= VALUES < 715827882 I0220 18:44:24.842839 24480 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet a7a069dabd5b4df28954b6833f23ab31. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:24.843217 24477 tablet_service.cc:1067] Processing CreateTablet for tablet 827aa98f727b4da6a5015391b4da1960 (table=test-workload [id=73578aeb8f9e4f9597b79f5737a60111]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1789569705 I0220 18:44:24.843353 24477 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 827aa98f727b4da6a5015391b4da1960. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:24.843749 24476 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet c0ca382ac98b46948b2b18c2d691a364. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:24.843827 24229 tablet_service.cc:1067] Processing CreateTablet for tablet 827aa98f727b4da6a5015391b4da1960 (table=test-workload [id=73578aeb8f9e4f9597b79f5737a60111]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1789569705 I0220 18:44:24.844023 24229 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 827aa98f727b4da6a5015391b4da1960. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:24.844126 24355 tablet_service.cc:1067] Processing CreateTablet for tablet 2ca7a33c93d548f49d2a4b1eeca470ca (table=test-workload [id=73578aeb8f9e4f9597b79f5737a60111]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1073741823 I0220 18:44:24.844295 24355 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 2ca7a33c93d548f49d2a4b1eeca470ca. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:24.847534 24353 tablet_service.cc:1067] Processing CreateTablet for tablet 827aa98f727b4da6a5015391b4da1960 (table=test-workload [id=73578aeb8f9e4f9597b79f5737a60111]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1789569705 I0220 18:44:24.847556 24230 tablet_service.cc:1067] Processing CreateTablet for tablet c5b2eb9c654f49df828ad8a096c62921 (table=test-workload [id=73578aeb8f9e4f9597b79f5737a60111]), partition=RANGE (key) PARTITION 1073741823 <= VALUES < 1431655764 I0220 18:44:24.847779 24357 tablet_service.cc:1067] Processing CreateTablet for tablet 116ca55e12164875ad0a87183fe531bc (table=test-workload [id=73578aeb8f9e4f9597b79f5737a60111]), partition=RANGE (key) PARTITION VALUES < 357913941 I0220 18:44:24.847790 24228 tablet_service.cc:1067] Processing CreateTablet for tablet c0ca382ac98b46948b2b18c2d691a364 (table=test-workload [id=73578aeb8f9e4f9597b79f5737a60111]), partition=RANGE (key) PARTITION VALUES >= 1789569705 I0220 18:44:24.847928 24228 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet c0ca382ac98b46948b2b18c2d691a364. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:24.847983 24354 tablet_service.cc:1067] Processing CreateTablet for tablet c5b2eb9c654f49df828ad8a096c62921 (table=test-workload [id=73578aeb8f9e4f9597b79f5737a60111]), partition=RANGE (key) PARTITION 1073741823 <= VALUES < 1431655764 I0220 18:44:24.848134 24353 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 827aa98f727b4da6a5015391b4da1960. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:24.848740 24479 tablet_service.cc:1067] Processing CreateTablet for tablet 2ca7a33c93d548f49d2a4b1eeca470ca (table=test-workload [id=73578aeb8f9e4f9597b79f5737a60111]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1073741823 I0220 18:44:24.848886 24479 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 2ca7a33c93d548f49d2a4b1eeca470ca. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:24.849545 24357 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 116ca55e12164875ad0a87183fe531bc. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:24.849726 24231 tablet_service.cc:1067] Processing CreateTablet for tablet 2ca7a33c93d548f49d2a4b1eeca470ca (table=test-workload [id=73578aeb8f9e4f9597b79f5737a60111]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1073741823 I0220 18:44:24.849879 24231 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 2ca7a33c93d548f49d2a4b1eeca470ca. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:24.851006 24565 tablet_bootstrap.cc:441] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa: Bootstrap starting. I0220 18:44:24.851291 24356 tablet_service.cc:1067] Processing CreateTablet for tablet a7a069dabd5b4df28954b6833f23ab31 (table=test-workload [id=73578aeb8f9e4f9597b79f5737a60111]), partition=RANGE (key) PARTITION 357913941 <= VALUES < 715827882 I0220 18:44:24.851336 24564 tablet_bootstrap.cc:441] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87: Bootstrap starting. I0220 18:44:24.851459 24232 tablet_service.cc:1067] Processing CreateTablet for tablet a7a069dabd5b4df28954b6833f23ab31 (table=test-workload [id=73578aeb8f9e4f9597b79f5737a60111]), partition=RANGE (key) PARTITION 357913941 <= VALUES < 715827882 I0220 18:44:24.851482 24356 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet a7a069dabd5b4df28954b6833f23ab31. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:24.851608 24232 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet a7a069dabd5b4df28954b6833f23ab31. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:24.851876 24566 tablet_bootstrap.cc:441] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap starting. I0220 18:44:24.853164 24354 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet c5b2eb9c654f49df828ad8a096c62921. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:24.853545 24230 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet c5b2eb9c654f49df828ad8a096c62921. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:24.853559 24566 tablet_bootstrap.cc:597] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de: No blocks or log segments found. Creating new log. I0220 18:44:24.854296 24566 log.cc:828] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de: Log is configured to *not* fsync() on all Append() calls I0220 18:44:24.856921 24564 tablet_bootstrap.cc:597] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87: No blocks or log segments found. Creating new log. I0220 18:44:24.857105 24566 tablet_bootstrap.cc:441] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de: No bootstrap required, opened a new log I0220 18:44:24.857383 24566 ts_tablet_manager.cc:1113] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s I0220 18:44:24.857619 24564 log.cc:828] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87: Log is configured to *not* fsync() on all Append() calls I0220 18:44:24.858727 24565 tablet_bootstrap.cc:597] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa: No blocks or log segments found. Creating new log. I0220 18:44:24.859042 24564 tablet_bootstrap.cc:441] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87: No bootstrap required, opened a new log I0220 18:44:24.859196 24564 ts_tablet_manager.cc:1113] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s I0220 18:44:24.859375 24565 log.cc:828] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa: Log is configured to *not* fsync() on all Append() calls I0220 18:44:24.859500 24566 raft_consensus.cc:346] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.859681 24566 raft_consensus.cc:372] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:24.859771 24566 raft_consensus.cc:726] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493be4abe83d46c1829e3d5cfc68a9de, State: Initialized, Role: FOLLOWER I0220 18:44:24.860285 24566 consensus_queue.cc:252] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.860581 24565 tablet_bootstrap.cc:441] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa: No bootstrap required, opened a new log I0220 18:44:24.860800 24295 heartbeater.cc:447] Master 127.23.135.190:45127 was elected leader, sending a full tablet report... I0220 18:44:24.860929 24565 ts_tablet_manager.cc:1113] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s I0220 18:44:24.861950 24564 raft_consensus.cc:346] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.862982 24564 raft_consensus.cc:372] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:24.863039 24564 raft_consensus.cc:726] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a15b6a57804d05ada89b15de898a87, State: Initialized, Role: FOLLOWER I0220 18:44:24.863467 24564 consensus_queue.cc:252] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.863519 24565 raft_consensus.cc:346] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } I0220 18:44:24.863567 24565 raft_consensus.cc:372] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:24.863600 24565 raft_consensus.cc:726] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bccadedc7a11414384f1d4875bf49aaa, State: Initialized, Role: FOLLOWER I0220 18:44:24.864100 24565 consensus_queue.cc:252] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } I0220 18:44:24.864581 24544 heartbeater.cc:447] Master 127.23.135.190:45127 was elected leader, sending a full tablet report... I0220 18:44:24.864614 24420 heartbeater.cc:447] Master 127.23.135.190:45127 was elected leader, sending a full tablet report... I0220 18:44:24.865994 24565 ts_tablet_manager.cc:1142] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s I0220 18:44:24.866253 24565 tablet_bootstrap.cc:441] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa: Bootstrap starting. I0220 18:44:24.867228 24566 ts_tablet_manager.cc:1142] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s I0220 18:44:24.867298 24565 tablet_bootstrap.cc:597] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa: No blocks or log segments found. Creating new log. I0220 18:44:24.868479 24566 tablet_bootstrap.cc:441] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap starting. I0220 18:44:24.868491 24564 ts_tablet_manager.cc:1142] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s I0220 18:44:24.868659 24564 tablet_bootstrap.cc:441] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87: Bootstrap starting. I0220 18:44:24.869385 24566 tablet_bootstrap.cc:597] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de: No blocks or log segments found. Creating new log. I0220 18:44:24.869491 24564 tablet_bootstrap.cc:597] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87: No blocks or log segments found. Creating new log. I0220 18:44:24.870400 24565 tablet_bootstrap.cc:441] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa: No bootstrap required, opened a new log I0220 18:44:24.870488 24565 ts_tablet_manager.cc:1113] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s I0220 18:44:24.870543 24566 tablet_bootstrap.cc:441] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de: No bootstrap required, opened a new log I0220 18:44:24.870630 24566 ts_tablet_manager.cc:1113] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s I0220 18:44:24.871003 24565 raft_consensus.cc:346] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.871083 24565 raft_consensus.cc:372] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:24.871098 24566 raft_consensus.cc:346] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.871136 24565 raft_consensus.cc:726] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bccadedc7a11414384f1d4875bf49aaa, State: Initialized, Role: FOLLOWER I0220 18:44:24.871201 24566 raft_consensus.cc:372] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:24.871256 24566 raft_consensus.cc:726] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493be4abe83d46c1829e3d5cfc68a9de, State: Initialized, Role: FOLLOWER I0220 18:44:24.871335 24565 consensus_queue.cc:252] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.871477 24566 consensus_queue.cc:252] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.871874 24565 ts_tablet_manager.cc:1142] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I0220 18:44:24.872043 24565 tablet_bootstrap.cc:441] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa: Bootstrap starting. I0220 18:44:24.872265 24564 tablet_bootstrap.cc:441] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87: No bootstrap required, opened a new log I0220 18:44:24.872372 24564 ts_tablet_manager.cc:1113] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s I0220 18:44:24.872874 24564 raft_consensus.cc:346] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.872956 24564 raft_consensus.cc:372] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:24.872958 24565 tablet_bootstrap.cc:597] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa: No blocks or log segments found. Creating new log. I0220 18:44:24.873021 24564 raft_consensus.cc:726] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a15b6a57804d05ada89b15de898a87, State: Initialized, Role: FOLLOWER I0220 18:44:24.873214 24570 raft_consensus.cc:482] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:44:24.873253 24564 consensus_queue.cc:252] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.873320 24570 raft_consensus.cc:504] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.873415 24564 ts_tablet_manager.cc:1142] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0220 18:44:24.873504 24564 tablet_bootstrap.cc:441] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87: Bootstrap starting. I0220 18:44:24.874107 24564 tablet_bootstrap.cc:597] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87: No blocks or log segments found. Creating new log. I0220 18:44:24.874300 24570 leader_election.cc:293] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:24.875063 24565 tablet_bootstrap.cc:441] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa: No bootstrap required, opened a new log I0220 18:44:24.875439 24565 ts_tablet_manager.cc:1113] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s I0220 18:44:24.875455 24566 ts_tablet_manager.cc:1142] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s I0220 18:44:24.875581 24566 tablet_bootstrap.cc:441] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap starting. I0220 18:44:24.875964 24565 raft_consensus.cc:346] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.876063 24565 raft_consensus.cc:372] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:24.876128 24565 raft_consensus.cc:726] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bccadedc7a11414384f1d4875bf49aaa, State: Initialized, Role: FOLLOWER I0220 18:44:24.876315 24565 consensus_queue.cc:252] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.876492 24566 tablet_bootstrap.cc:597] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de: No blocks or log segments found. Creating new log. I0220 18:44:24.876528 24565 ts_tablet_manager.cc:1142] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0220 18:44:24.877121 24564 tablet_bootstrap.cc:441] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87: No bootstrap required, opened a new log I0220 18:44:24.877188 24564 ts_tablet_manager.cc:1113] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s I0220 18:44:24.877545 24565 tablet_bootstrap.cc:441] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa: Bootstrap starting. I0220 18:44:24.878319 24565 tablet_bootstrap.cc:597] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa: No blocks or log segments found. Creating new log. I0220 18:44:24.879081 24566 tablet_bootstrap.cc:441] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de: No bootstrap required, opened a new log I0220 18:44:24.879503 24564 raft_consensus.cc:346] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } I0220 18:44:24.879535 24564 raft_consensus.cc:372] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:24.879550 24564 raft_consensus.cc:726] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a15b6a57804d05ada89b15de898a87, State: Initialized, Role: FOLLOWER I0220 18:44:24.879730 24566 ts_tablet_manager.cc:1113] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s I0220 18:44:24.880180 24566 raft_consensus.cc:346] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.880328 24564 consensus_queue.cc:252] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } I0220 18:44:24.880429 24377 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "c0ca382ac98b46948b2b18c2d691a364" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" is_pre_election: true I0220 18:44:24.880578 24564 ts_tablet_manager.cc:1142] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0220 18:44:24.880682 24564 tablet_bootstrap.cc:441] T a7a069dabd5b4df28954b6833f23ab31 P c8a15b6a57804d05ada89b15de898a87: Bootstrap starting. I0220 18:44:24.881487 24564 tablet_bootstrap.cc:597] T a7a069dabd5b4df28954b6833f23ab31 P c8a15b6a57804d05ada89b15de898a87: No blocks or log segments found. Creating new log. W0220 18:44:24.881196 24183 leader_election.cc:346] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621): Illegal state: must be running to vote when last-logged opid is not known I0220 18:44:24.881960 24566 raft_consensus.cc:372] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:24.881983 24566 raft_consensus.cc:726] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493be4abe83d46c1829e3d5cfc68a9de, State: Initialized, Role: FOLLOWER I0220 18:44:24.882153 24566 consensus_queue.cc:252] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.882416 24566 ts_tablet_manager.cc:1142] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s I0220 18:44:24.882905 24501 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "c0ca382ac98b46948b2b18c2d691a364" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" is_pre_election: true I0220 18:44:24.882975 24566 tablet_bootstrap.cc:441] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap starting. W0220 18:44:24.883343 24183 leader_election.cc:346] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803): Illegal state: must be running to vote when last-logged opid is not known I0220 18:44:24.883409 24183 leader_election.cc:307] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 493be4abe83d46c1829e3d5cfc68a9de; no voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87 I0220 18:44:24.883513 24576 raft_consensus.cc:482] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:44:24.883671 24576 raft_consensus.cc:504] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.883935 24570 raft_consensus.cc:2675] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0220 18:44:24.883981 24565 tablet_bootstrap.cc:441] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa: No bootstrap required, opened a new log I0220 18:44:24.884052 24565 ts_tablet_manager.cc:1113] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s I0220 18:44:24.884510 24565 raft_consensus.cc:346] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.884555 24565 raft_consensus.cc:372] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:24.884572 24565 raft_consensus.cc:726] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bccadedc7a11414384f1d4875bf49aaa, State: Initialized, Role: FOLLOWER I0220 18:44:24.885010 24566 tablet_bootstrap.cc:597] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de: No blocks or log segments found. Creating new log. I0220 18:44:24.885944 24564 tablet_bootstrap.cc:441] T a7a069dabd5b4df28954b6833f23ab31 P c8a15b6a57804d05ada89b15de898a87: No bootstrap required, opened a new log I0220 18:44:24.886034 24564 ts_tablet_manager.cc:1113] T a7a069dabd5b4df28954b6833f23ab31 P c8a15b6a57804d05ada89b15de898a87: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s I0220 18:44:24.886433 24564 raft_consensus.cc:346] T a7a069dabd5b4df28954b6833f23ab31 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.886476 24564 raft_consensus.cc:372] T a7a069dabd5b4df28954b6833f23ab31 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:24.886668 24565 consensus_queue.cc:252] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.885004 24576 leader_election.cc:293] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559), c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621) I0220 18:44:24.887938 24565 ts_tablet_manager.cc:1142] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s I0220 18:44:24.888319 24565 tablet_bootstrap.cc:441] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa: Bootstrap starting. I0220 18:44:24.888442 24252 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "c5b2eb9c654f49df828ad8a096c62921" candidate_uuid: "bccadedc7a11414384f1d4875bf49aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493be4abe83d46c1829e3d5cfc68a9de" is_pre_election: true I0220 18:44:24.888689 24566 tablet_bootstrap.cc:441] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de: No bootstrap required, opened a new log I0220 18:44:24.888751 24566 ts_tablet_manager.cc:1113] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s I0220 18:44:24.889165 24566 raft_consensus.cc:346] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } W0220 18:44:24.889030 24433 leader_election.cc:346] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Illegal state: must be running to vote when last-logged opid is not known I0220 18:44:24.889921 24565 tablet_bootstrap.cc:597] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa: No blocks or log segments found. Creating new log. I0220 18:44:24.889398 24377 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "c5b2eb9c654f49df828ad8a096c62921" candidate_uuid: "bccadedc7a11414384f1d4875bf49aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" is_pre_election: true I0220 18:44:24.890328 24564 raft_consensus.cc:726] T a7a069dabd5b4df28954b6833f23ab31 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a15b6a57804d05ada89b15de898a87, State: Initialized, Role: FOLLOWER I0220 18:44:24.890475 24564 consensus_queue.cc:252] T a7a069dabd5b4df28954b6833f23ab31 P c8a15b6a57804d05ada89b15de898a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } W0220 18:44:24.890611 24432 leader_election.cc:346] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621): Illegal state: must be running to vote when last-logged opid is not known I0220 18:44:24.890676 24432 leader_election.cc:307] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bccadedc7a11414384f1d4875bf49aaa; no voters: 493be4abe83d46c1829e3d5cfc68a9de, c8a15b6a57804d05ada89b15de898a87 I0220 18:44:24.890722 24564 ts_tablet_manager.cc:1142] T a7a069dabd5b4df28954b6833f23ab31 P c8a15b6a57804d05ada89b15de898a87: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s I0220 18:44:24.890844 24564 tablet_bootstrap.cc:441] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87: Bootstrap starting. I0220 18:44:24.890942 24576 raft_consensus.cc:2675] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0220 18:44:24.891607 24564 tablet_bootstrap.cc:597] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87: No blocks or log segments found. Creating new log. I0220 18:44:24.891678 24566 raft_consensus.cc:372] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:24.891706 24566 raft_consensus.cc:726] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493be4abe83d46c1829e3d5cfc68a9de, State: Initialized, Role: FOLLOWER I0220 18:44:24.891853 24566 consensus_queue.cc:252] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.892082 24566 ts_tablet_manager.cc:1142] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s I0220 18:44:24.892200 24566 tablet_bootstrap.cc:441] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap starting. I0220 18:44:24.892740 24565 tablet_bootstrap.cc:441] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa: No bootstrap required, opened a new log I0220 18:44:24.892817 24565 ts_tablet_manager.cc:1113] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s I0220 18:44:24.893132 24566 tablet_bootstrap.cc:597] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de: No blocks or log segments found. Creating new log. I0220 18:44:24.893271 24565 raft_consensus.cc:346] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.893301 24565 raft_consensus.cc:372] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:24.893316 24565 raft_consensus.cc:726] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bccadedc7a11414384f1d4875bf49aaa, State: Initialized, Role: FOLLOWER I0220 18:44:24.893455 24565 consensus_queue.cc:252] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.893828 24565 ts_tablet_manager.cc:1142] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0220 18:44:24.893981 24565 tablet_bootstrap.cc:441] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa: Bootstrap starting. I0220 18:44:24.894523 24564 tablet_bootstrap.cc:441] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87: No bootstrap required, opened a new log I0220 18:44:24.894629 24564 ts_tablet_manager.cc:1113] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s I0220 18:44:24.894660 24566 tablet_bootstrap.cc:441] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de: No bootstrap required, opened a new log I0220 18:44:24.894717 24566 ts_tablet_manager.cc:1113] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I0220 18:44:24.894806 24565 tablet_bootstrap.cc:597] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa: No blocks or log segments found. Creating new log. I0220 18:44:24.895076 24564 raft_consensus.cc:346] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.895113 24564 raft_consensus.cc:372] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:24.895117 24566 raft_consensus.cc:346] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.895128 24564 raft_consensus.cc:726] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a15b6a57804d05ada89b15de898a87, State: Initialized, Role: FOLLOWER I0220 18:44:24.895136 24566 raft_consensus.cc:372] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:24.895150 24566 raft_consensus.cc:726] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493be4abe83d46c1829e3d5cfc68a9de, State: Initialized, Role: FOLLOWER I0220 18:44:24.895275 24564 consensus_queue.cc:252] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.895284 24566 consensus_queue.cc:252] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.895471 24564 ts_tablet_manager.cc:1142] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I0220 18:44:24.895607 24564 tablet_bootstrap.cc:441] T c0ca382ac98b46948b2b18c2d691a364 P c8a15b6a57804d05ada89b15de898a87: Bootstrap starting. I0220 18:44:24.896031 24566 ts_tablet_manager.cc:1142] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0220 18:44:24.896340 24565 tablet_bootstrap.cc:441] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa: No bootstrap required, opened a new log I0220 18:44:24.896435 24564 tablet_bootstrap.cc:597] T c0ca382ac98b46948b2b18c2d691a364 P c8a15b6a57804d05ada89b15de898a87: No blocks or log segments found. Creating new log. I0220 18:44:24.896517 24566 tablet_bootstrap.cc:441] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap starting. I0220 18:44:24.897347 24566 tablet_bootstrap.cc:597] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de: No blocks or log segments found. Creating new log. I0220 18:44:24.897388 24564 tablet_bootstrap.cc:441] T c0ca382ac98b46948b2b18c2d691a364 P c8a15b6a57804d05ada89b15de898a87: No bootstrap required, opened a new log I0220 18:44:24.897464 24565 ts_tablet_manager.cc:1113] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s I0220 18:44:24.897470 24564 ts_tablet_manager.cc:1113] T c0ca382ac98b46948b2b18c2d691a364 P c8a15b6a57804d05ada89b15de898a87: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s I0220 18:44:24.897891 24565 raft_consensus.cc:346] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.897891 24564 raft_consensus.cc:346] T c0ca382ac98b46948b2b18c2d691a364 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.897918 24564 raft_consensus.cc:372] T c0ca382ac98b46948b2b18c2d691a364 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:24.897933 24564 raft_consensus.cc:726] T c0ca382ac98b46948b2b18c2d691a364 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a15b6a57804d05ada89b15de898a87, State: Initialized, Role: FOLLOWER I0220 18:44:24.897967 24565 raft_consensus.cc:372] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:24.897992 24565 raft_consensus.cc:726] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bccadedc7a11414384f1d4875bf49aaa, State: Initialized, Role: FOLLOWER I0220 18:44:24.898072 24564 consensus_queue.cc:252] T c0ca382ac98b46948b2b18c2d691a364 P c8a15b6a57804d05ada89b15de898a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.898108 24565 consensus_queue.cc:252] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.898296 24565 ts_tablet_manager.cc:1142] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0220 18:44:24.897512 24570 raft_consensus.cc:482] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:44:24.898485 24570 raft_consensus.cc:504] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.898746 24570 leader_election.cc:293] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:24.899329 24566 tablet_bootstrap.cc:441] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de: No bootstrap required, opened a new log I0220 18:44:24.899341 24377 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "2ca7a33c93d548f49d2a4b1eeca470ca" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" is_pre_election: true I0220 18:44:24.899453 24566 ts_tablet_manager.cc:1113] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I0220 18:44:24.899488 24377 raft_consensus.cc:2414] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 493be4abe83d46c1829e3d5cfc68a9de in term 0. I0220 18:44:24.899881 24183 leader_election.cc:307] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493be4abe83d46c1829e3d5cfc68a9de, c8a15b6a57804d05ada89b15de898a87; no voters: I0220 18:44:24.899930 24566 raft_consensus.cc:346] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } I0220 18:44:24.899952 24566 raft_consensus.cc:372] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:24.899966 24566 raft_consensus.cc:726] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493be4abe83d46c1829e3d5cfc68a9de, State: Initialized, Role: FOLLOWER I0220 18:44:24.900121 24570 raft_consensus.cc:2730] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Leader pre-election won for term 1 I0220 18:44:24.900130 24566 consensus_queue.cc:252] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } I0220 18:44:24.900162 24570 raft_consensus.cc:482] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0220 18:44:24.900185 24570 raft_consensus.cc:2983] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:24.900454 24564 ts_tablet_manager.cc:1142] T c0ca382ac98b46948b2b18c2d691a364 P c8a15b6a57804d05ada89b15de898a87: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s I0220 18:44:24.900643 24566 ts_tablet_manager.cc:1142] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0220 18:44:24.901058 24570 raft_consensus.cc:504] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.901280 24570 leader_election.cc:293] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 1 election: Requested vote from peers c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:24.901500 24501 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "2ca7a33c93d548f49d2a4b1eeca470ca" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" is_pre_election: true I0220 18:44:24.901587 24377 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "2ca7a33c93d548f49d2a4b1eeca470ca" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" I0220 18:44:24.901635 24501 raft_consensus.cc:2414] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 493be4abe83d46c1829e3d5cfc68a9de in term 0. I0220 18:44:24.901645 24377 raft_consensus.cc:2983] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:24.901648 24500 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "2ca7a33c93d548f49d2a4b1eeca470ca" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" I0220 18:44:24.901697 24500 raft_consensus.cc:2983] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:24.902479 24377 raft_consensus.cc:2414] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 493be4abe83d46c1829e3d5cfc68a9de in term 1. I0220 18:44:24.902479 24500 raft_consensus.cc:2414] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 493be4abe83d46c1829e3d5cfc68a9de in term 1. I0220 18:44:24.902736 24183 leader_election.cc:307] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493be4abe83d46c1829e3d5cfc68a9de, bccadedc7a11414384f1d4875bf49aaa; no voters: I0220 18:44:24.902863 24570 raft_consensus.cc:2730] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Leader election won for term 1 I0220 18:44:24.902933 24570 raft_consensus.cc:686] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [term 1 LEADER]: Becoming Leader. State: Replica: 493be4abe83d46c1829e3d5cfc68a9de, State: Running, Role: LEADER I0220 18:44:24.903204 24570 consensus_queue.cc:229] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.905890 24128 catalog_manager.cc:4173] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de reported cstate change: term changed from 0 to 1, leader changed from to 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129). New cstate: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } health_report { overall_health: UNKNOWN } } } I0220 18:44:24.916332 24576 raft_consensus.cc:482] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:44:24.916455 24576 raft_consensus.cc:504] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.916754 24576 leader_election.cc:293] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559), c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621) I0220 18:44:24.917160 24252 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "c0ca382ac98b46948b2b18c2d691a364" candidate_uuid: "bccadedc7a11414384f1d4875bf49aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493be4abe83d46c1829e3d5cfc68a9de" is_pre_election: true I0220 18:44:24.917183 24377 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "c0ca382ac98b46948b2b18c2d691a364" candidate_uuid: "bccadedc7a11414384f1d4875bf49aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" is_pre_election: true I0220 18:44:24.917264 24377 raft_consensus.cc:2414] T c0ca382ac98b46948b2b18c2d691a364 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bccadedc7a11414384f1d4875bf49aaa in term 0. I0220 18:44:24.917275 24252 raft_consensus.cc:2414] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bccadedc7a11414384f1d4875bf49aaa in term 0. I0220 18:44:24.917577 24433 leader_election.cc:307] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493be4abe83d46c1829e3d5cfc68a9de, bccadedc7a11414384f1d4875bf49aaa; no voters: I0220 18:44:24.917713 24576 raft_consensus.cc:2730] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Leader pre-election won for term 1 I0220 18:44:24.917747 24576 raft_consensus.cc:482] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0220 18:44:24.917762 24576 raft_consensus.cc:2983] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:24.918547 24576 raft_consensus.cc:504] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.918785 24576 leader_election.cc:293] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 1 election: Requested vote from peers 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559), c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621) I0220 18:44:24.919085 24252 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "c0ca382ac98b46948b2b18c2d691a364" candidate_uuid: "bccadedc7a11414384f1d4875bf49aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493be4abe83d46c1829e3d5cfc68a9de" I0220 18:44:24.919137 24252 raft_consensus.cc:2983] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:24.919145 24377 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "c0ca382ac98b46948b2b18c2d691a364" candidate_uuid: "bccadedc7a11414384f1d4875bf49aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" I0220 18:44:24.919198 24377 raft_consensus.cc:2983] T c0ca382ac98b46948b2b18c2d691a364 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:24.919904 24252 raft_consensus.cc:2414] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bccadedc7a11414384f1d4875bf49aaa in term 1. I0220 18:44:24.919909 24377 raft_consensus.cc:2414] T c0ca382ac98b46948b2b18c2d691a364 P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bccadedc7a11414384f1d4875bf49aaa in term 1. I0220 18:44:24.920192 24432 leader_election.cc:307] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87; no voters: I0220 18:44:24.920341 24576 raft_consensus.cc:2730] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Leader election won for term 1 I0220 18:44:24.920420 24576 raft_consensus.cc:686] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [term 1 LEADER]: Becoming Leader. State: Replica: bccadedc7a11414384f1d4875bf49aaa, State: Running, Role: LEADER I0220 18:44:24.920702 24576 consensus_queue.cc:229] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.923307 24128 catalog_manager.cc:4173] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa reported cstate change: term changed from 0 to 1, leader changed from to bccadedc7a11414384f1d4875bf49aaa (127.23.135.131). New cstate: current_term: 1 leader_uuid: "bccadedc7a11414384f1d4875bf49aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } health_report { overall_health: HEALTHY } } } I0220 18:44:24.931838 24570 raft_consensus.cc:482] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:44:24.931941 24570 raft_consensus.cc:504] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } I0220 18:44:24.932199 24570 leader_election.cc:293] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:24.932651 24500 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "116ca55e12164875ad0a87183fe531bc" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" is_pre_election: true I0220 18:44:24.932696 24377 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "116ca55e12164875ad0a87183fe531bc" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" is_pre_election: true I0220 18:44:24.932734 24500 raft_consensus.cc:2414] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 493be4abe83d46c1829e3d5cfc68a9de in term 0. I0220 18:44:24.932776 24377 raft_consensus.cc:2414] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 493be4abe83d46c1829e3d5cfc68a9de in term 0. I0220 18:44:24.933029 24183 leader_election.cc:307] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493be4abe83d46c1829e3d5cfc68a9de, bccadedc7a11414384f1d4875bf49aaa; no voters: I0220 18:44:24.933190 24570 raft_consensus.cc:2730] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Leader pre-election won for term 1 I0220 18:44:24.933219 24570 raft_consensus.cc:482] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0220 18:44:24.933235 24570 raft_consensus.cc:2983] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Advancing to term 1 W0220 18:44:24.933503 24421 tablet.cc:1810] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87: Can't schedule compaction. Clean time has not been advanced past its initial value. W0220 18:44:24.933645 24421 tablet_replica_mm_ops.cc:302] Log GC is disabled (check --enable_log_gc) I0220 18:44:24.934128 24570 raft_consensus.cc:504] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } I0220 18:44:24.934347 24570 leader_election.cc:293] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 1 election: Requested vote from peers c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:24.934715 24500 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "116ca55e12164875ad0a87183fe531bc" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" I0220 18:44:24.934736 24377 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "116ca55e12164875ad0a87183fe531bc" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" I0220 18:44:24.934787 24377 raft_consensus.cc:2983] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:24.934788 24500 raft_consensus.cc:2983] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:24.935408 24377 raft_consensus.cc:2414] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 493be4abe83d46c1829e3d5cfc68a9de in term 1. I0220 18:44:24.935413 24500 raft_consensus.cc:2414] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 493be4abe83d46c1829e3d5cfc68a9de in term 1. I0220 18:44:24.935684 24183 leader_election.cc:307] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493be4abe83d46c1829e3d5cfc68a9de, c8a15b6a57804d05ada89b15de898a87; no voters: I0220 18:44:24.935814 24570 raft_consensus.cc:2730] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Leader election won for term 1 I0220 18:44:24.935884 24570 raft_consensus.cc:686] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [term 1 LEADER]: Becoming Leader. State: Replica: 493be4abe83d46c1829e3d5cfc68a9de, State: Running, Role: LEADER I0220 18:44:24.936026 24570 consensus_queue.cc:229] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } I0220 18:44:24.937505 24128 catalog_manager.cc:4173] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de reported cstate change: term changed from 0 to 1, leader changed from to 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129). New cstate: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } health_report { overall_health: HEALTHY } } } I0220 18:44:24.941610 24571 raft_consensus.cc:482] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:44:24.941694 24571 raft_consensus.cc:504] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.942484 24571 leader_election.cc:293] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:24.942538 24570 raft_consensus.cc:482] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:44:24.942621 24570 raft_consensus.cc:504] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.942878 24570 leader_election.cc:293] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:24.943349 24377 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "827aa98f727b4da6a5015391b4da1960" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" is_pre_election: true I0220 18:44:24.943377 24500 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "827aa98f727b4da6a5015391b4da1960" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" is_pre_election: true I0220 18:44:24.943429 24377 raft_consensus.cc:2414] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 493be4abe83d46c1829e3d5cfc68a9de in term 0. I0220 18:44:24.943444 24500 raft_consensus.cc:2414] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 493be4abe83d46c1829e3d5cfc68a9de in term 0. I0220 18:44:24.943722 24183 leader_election.cc:307] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493be4abe83d46c1829e3d5cfc68a9de, c8a15b6a57804d05ada89b15de898a87; no voters: I0220 18:44:24.943845 24570 raft_consensus.cc:2730] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Leader pre-election won for term 1 I0220 18:44:24.943902 24570 raft_consensus.cc:482] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0220 18:44:24.943958 24570 raft_consensus.cc:2983] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:24.944219 24576 raft_consensus.cc:482] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:44:24.944332 24576 raft_consensus.cc:504] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.944873 24570 raft_consensus.cc:504] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.944973 24377 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "a7a069dabd5b4df28954b6833f23ab31" candidate_uuid: "bccadedc7a11414384f1d4875bf49aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" is_pre_election: true I0220 18:44:24.945034 24377 raft_consensus.cc:2414] T a7a069dabd5b4df28954b6833f23ab31 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bccadedc7a11414384f1d4875bf49aaa in term 0. I0220 18:44:24.945168 24570 leader_election.cc:293] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 1 election: Requested vote from peers c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:24.945327 24432 leader_election.cc:307] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87; no voters: I0220 18:44:24.945482 24572 raft_consensus.cc:2730] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Leader pre-election won for term 1 I0220 18:44:24.945506 24572 raft_consensus.cc:482] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0220 18:44:24.945523 24572 raft_consensus.cc:2983] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:24.946280 24572 raft_consensus.cc:504] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.946478 24572 leader_election.cc:293] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 1 election: Requested vote from peers c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621), 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559) I0220 18:44:24.946748 24377 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "a7a069dabd5b4df28954b6833f23ab31" candidate_uuid: "bccadedc7a11414384f1d4875bf49aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" I0220 18:44:24.946795 24377 raft_consensus.cc:2983] T a7a069dabd5b4df28954b6833f23ab31 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:24.946883 24252 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "a7a069dabd5b4df28954b6833f23ab31" candidate_uuid: "bccadedc7a11414384f1d4875bf49aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493be4abe83d46c1829e3d5cfc68a9de" I0220 18:44:24.946926 24252 raft_consensus.cc:2983] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:24.947562 24377 raft_consensus.cc:2414] T a7a069dabd5b4df28954b6833f23ab31 P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bccadedc7a11414384f1d4875bf49aaa in term 1. I0220 18:44:24.947809 24252 raft_consensus.cc:2414] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bccadedc7a11414384f1d4875bf49aaa in term 1. I0220 18:44:24.947856 24432 leader_election.cc:307] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87; no voters: I0220 18:44:24.947914 24251 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "827aa98f727b4da6a5015391b4da1960" candidate_uuid: "c8a15b6a57804d05ada89b15de898a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493be4abe83d46c1829e3d5cfc68a9de" is_pre_election: true I0220 18:44:24.947973 24572 raft_consensus.cc:2730] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Leader election won for term 1 I0220 18:44:24.948025 24251 raft_consensus.cc:2339] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c8a15b6a57804d05ada89b15de898a87 in current term 1: Already voted for candidate 493be4abe83d46c1829e3d5cfc68a9de in this term. I0220 18:44:24.948128 24572 raft_consensus.cc:686] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [term 1 LEADER]: Becoming Leader. State: Replica: bccadedc7a11414384f1d4875bf49aaa, State: Running, Role: LEADER I0220 18:44:24.948267 24572 consensus_queue.cc:229] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.948637 24500 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "827aa98f727b4da6a5015391b4da1960" candidate_uuid: "c8a15b6a57804d05ada89b15de898a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" is_pre_election: true I0220 18:44:24.948695 24500 raft_consensus.cc:2414] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a15b6a57804d05ada89b15de898a87 in term 0. I0220 18:44:24.948837 24377 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "827aa98f727b4da6a5015391b4da1960" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" I0220 18:44:24.948881 24377 raft_consensus.cc:2983] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:24.948954 24308 leader_election.cc:307] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87; no voters: 493be4abe83d46c1829e3d5cfc68a9de I0220 18:44:24.949568 24377 raft_consensus.cc:2414] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 493be4abe83d46c1829e3d5cfc68a9de in term 1. I0220 18:44:24.949640 24571 raft_consensus.cc:2690] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won I0220 18:44:24.949743 24128 catalog_manager.cc:4173] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa reported cstate change: term changed from 0 to 1, leader changed from to bccadedc7a11414384f1d4875bf49aaa (127.23.135.131). New cstate: current_term: 1 leader_uuid: "bccadedc7a11414384f1d4875bf49aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } health_report { overall_health: HEALTHY } } } I0220 18:44:24.949822 24183 leader_election.cc:307] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493be4abe83d46c1829e3d5cfc68a9de, c8a15b6a57804d05ada89b15de898a87; no voters: I0220 18:44:24.949872 24501 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "827aa98f727b4da6a5015391b4da1960" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" I0220 18:44:24.949935 24501 raft_consensus.cc:2983] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:24.949956 24570 raft_consensus.cc:2730] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Leader election won for term 1 I0220 18:44:24.950017 24570 raft_consensus.cc:686] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 LEADER]: Becoming Leader. State: Replica: 493be4abe83d46c1829e3d5cfc68a9de, State: Running, Role: LEADER I0220 18:44:24.950145 24570 consensus_queue.cc:229] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.950683 24501 raft_consensus.cc:2414] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 493be4abe83d46c1829e3d5cfc68a9de in term 1. I0220 18:44:24.951071 24576 leader_election.cc:293] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621), 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559) I0220 18:44:24.951139 24251 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "a7a069dabd5b4df28954b6833f23ab31" candidate_uuid: "bccadedc7a11414384f1d4875bf49aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493be4abe83d46c1829e3d5cfc68a9de" is_pre_election: true I0220 18:44:24.951200 24251 raft_consensus.cc:2322] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate bccadedc7a11414384f1d4875bf49aaa in term 1. Re-sending same reply. I0220 18:44:24.951678 24130 catalog_manager.cc:4173] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de reported cstate change: term changed from 0 to 1, leader changed from to 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129). New cstate: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } health_report { overall_health: UNKNOWN } } } I0220 18:44:24.978798 24571 raft_consensus.cc:482] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:44:24.978930 24571 raft_consensus.cc:504] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.979267 24571 leader_election.cc:293] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:24.979802 24251 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "c5b2eb9c654f49df828ad8a096c62921" candidate_uuid: "c8a15b6a57804d05ada89b15de898a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493be4abe83d46c1829e3d5cfc68a9de" is_pre_election: true I0220 18:44:24.979810 24501 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "c5b2eb9c654f49df828ad8a096c62921" candidate_uuid: "c8a15b6a57804d05ada89b15de898a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" is_pre_election: true I0220 18:44:24.979914 24501 raft_consensus.cc:2414] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a15b6a57804d05ada89b15de898a87 in term 0. I0220 18:44:24.979916 24251 raft_consensus.cc:2414] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a15b6a57804d05ada89b15de898a87 in term 0. I0220 18:44:24.980255 24308 leader_election.cc:307] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87; no voters: I0220 18:44:24.980394 24571 raft_consensus.cc:2730] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0220 18:44:24.980432 24571 raft_consensus.cc:482] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0220 18:44:24.980450 24571 raft_consensus.cc:2983] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:24.981305 24571 raft_consensus.cc:504] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.981540 24571 leader_election.cc:293] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 1 election: Requested vote from peers 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:24.981614 24570 consensus_queue.cc:1005] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0220 18:44:24.981842 24251 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "c5b2eb9c654f49df828ad8a096c62921" candidate_uuid: "c8a15b6a57804d05ada89b15de898a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493be4abe83d46c1829e3d5cfc68a9de" I0220 18:44:24.981928 24501 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "c5b2eb9c654f49df828ad8a096c62921" candidate_uuid: "c8a15b6a57804d05ada89b15de898a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" I0220 18:44:24.981940 24251 raft_consensus.cc:2983] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:24.981986 24501 raft_consensus.cc:2983] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:24.982764 24251 raft_consensus.cc:2414] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a15b6a57804d05ada89b15de898a87 in term 1. I0220 18:44:24.982828 24501 raft_consensus.cc:2414] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a15b6a57804d05ada89b15de898a87 in term 1. I0220 18:44:24.983044 24309 leader_election.cc:307] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493be4abe83d46c1829e3d5cfc68a9de, c8a15b6a57804d05ada89b15de898a87; no voters: I0220 18:44:24.983160 24571 raft_consensus.cc:2730] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Leader election won for term 1 I0220 18:44:24.983319 24571 raft_consensus.cc:686] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [term 1 LEADER]: Becoming Leader. State: Replica: c8a15b6a57804d05ada89b15de898a87, State: Running, Role: LEADER I0220 18:44:24.983613 24571 consensus_queue.cc:229] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:24.986114 24130 catalog_manager.cc:4173] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 reported cstate change: term changed from 0 to 1, leader changed from to c8a15b6a57804d05ada89b15de898a87 (127.23.135.130). New cstate: current_term: 1 leader_uuid: "c8a15b6a57804d05ada89b15de898a87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } health_report { overall_health: UNKNOWN } } } I0220 18:44:24.986789 24591 consensus_queue.cc:1005] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [LEADER]: Connected to new peer: Peer: permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0220 18:44:25.000296 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-3/data --block_manager=log --rpc_bind_addresses=127.23.135.132:0 --local_ip_for_outbound_sockets=127.23.135.132 --webserver_interface=127.23.135.132 --webserver_port=0 --tserver_master_addrs=127.23.135.190:45127 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:41009 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --follower_unavailable_considered_failed_sec=2 --enable_log_gc=false with env {} I0220 18:44:25.003083 24576 consensus_queue.cc:1005] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0220 18:44:25.005762 24572 consensus_queue.cc:1005] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0220 18:44:25.019258 24591 consensus_queue.cc:1005] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0220 18:44:25.023352 24591 consensus_queue.cc:1005] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [LEADER]: Connected to new peer: Peer: permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0220 18:44:25.037415 24545 tablet.cc:1810] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa: Can't schedule compaction. Clean time has not been advanced past its initial value. W0220 18:44:25.037540 24545 tablet_replica_mm_ops.cc:302] Log GC is disabled (check --enable_log_gc) I0220 18:44:25.041393 24501 raft_consensus.cc:1210] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Refusing update from remote peer c8a15b6a57804d05ada89b15de898a87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0220 18:44:25.042060 24599 consensus_queue.cc:1005] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0220 18:44:25.042361 24251 raft_consensus.cc:1210] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Refusing update from remote peer c8a15b6a57804d05ada89b15de898a87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0220 18:44:25.042807 24599 consensus_queue.cc:1005] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0220 18:44:25.045964 24614 consensus_queue.cc:1005] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0220 18:44:25.050045 24501 raft_consensus.cc:1210] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Refusing update from remote peer 493be4abe83d46c1829e3d5cfc68a9de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0220 18:44:25.051369 24572 consensus_queue.cc:1005] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0220 18:44:25.052655 24377 raft_consensus.cc:1210] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Refusing update from remote peer 493be4abe83d46c1829e3d5cfc68a9de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0220 18:44:25.053411 24570 consensus_queue.cc:1005] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [LEADER]: Connected to new peer: Peer: permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0220 18:44:25.054837 24570 consensus_queue.cc:1005] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0220 18:44:25.062464 24606 mvcc.cc:203] Tried to move safe_time back from 6480790589603704832 to 6480790589374390272. Current Snapshot: MvccSnapshot[committed={T|T < 6480790589603704832}] W0220 18:44:25.065181 24296 tablet_replica_mm_ops.cc:302] Log GC is disabled (check --enable_log_gc) I0220 18:44:25.066154 24602 mvcc.cc:203] Tried to move safe_time back from 6480790589603704832 to 6480790589374390272. Current Snapshot: MvccSnapshot[committed={T|T < 6480790589603704832 or (T in {6480790589603704832})}] I0220 18:44:25.083190 24604 mvcc.cc:203] Tried to move safe_time back from 6480790589756866560 to 6480790589374390272. Current Snapshot: MvccSnapshot[committed={T|T < 6480790589756866560 or (T in {6480790589756866560})}] WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:44:25.094202 24611 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:44:25.094537 24611 flags.cc:403] Enabled unsafe flag: --enable_log_gc=false W0220 18:44:25.094599 24611 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:44:25.095913 24611 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:44:25.096055 24611 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.132 I0220 18:44:25.098016 24611 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:41009 --ntp_initial_sync_wait_secs=10 --time_source=builtin --follower_unavailable_considered_failed_sec=2 --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-3/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.132:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-3/data/info.pb --webserver_interface=127.23.135.132 --webserver_port=0 --enable_log_gc=false --tserver_master_addrs=127.23.135.190:45127 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.24611 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.132 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:44:25.098584 24611 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:44:25.099365 24611 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:44:25.106168 24611 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:44:25.107251 24611 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:44:25.111560 24611 hybrid_clock.cc:552] HybridClock initialized: now 1582224265111493 us; error 64 us; skew 500 ppm I0220 18:44:25.112318 24611 fs_manager.cc:259] Metadata directory not provided I0220 18:44:25.112359 24611 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:44:25.112571 24611 server_base.cc:502] Could not load existing FS layout: Not found: could not find a healthy instance file I0220 18:44:25.112679 24611 server_base.cc:503] Attempting to create new FS layout instead I0220 18:44:25.114575 24611 fs_manager.cc:613] Generated new instance metadata in path /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-3/data/instance: uuid: "609fb788e999458bb50fe5ede9fbe320" format_stamp: "Formatted at 2020-02-20 18:44:25 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:25.114972 24611 fs_manager.cc:613] Generated new instance metadata in path /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-3/wal/instance: uuid: "609fb788e999458bb50fe5ede9fbe320" format_stamp: "Formatted at 2020-02-20 18:44:25 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:25.117028 24611 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-3/data/data/block_manager_instance: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-3/data/data/block_manager_instance: No such file or directory (error 2) I0220 18:44:25.124171 24611 fs_manager.cc:514] Time spent creating directory manager: real 0.009s user 0.003s sys 0.004s I0220 18:44:25.168610 24611 fs_manager.cc:430] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0220 18:44:25.168722 24611 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-3/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-3/wal uuid: "609fb788e999458bb50fe5ede9fbe320" format_stamp: "Formatted at 2020-02-20 18:44:25 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:25.168824 24611 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-3/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-3/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:44:25.199296 24611 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:44:25.200932 24611 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0220 18:44:25.201114 24611 ts_tablet_manager.cc:415] Registered 0 tablets I0220 18:44:25.209527 24611 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.132:45837 I0220 18:44:25.210798 24611 webserver.cc:382] Webserver started at http://127.23.135.132:35635/ using document root and password file I0220 18:44:25.211608 24611 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-3/data/info.pb I0220 18:44:25.220504 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 24611 I0220 18:44:25.224165 24772 heartbeater.cc:322] Connected to a master server at 127.23.135.190:45127 I0220 18:44:25.224357 24772 heartbeater.cc:406] Registering TS with master... I0220 18:44:25.224719 24772 heartbeater.cc:455] Master 127.23.135.190:45127 requested a full tablet report, sending... I0220 18:44:25.225477 24128 ts_manager.cc:192] Registered new tserver with Master: 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132:45837) I0220 18:44:25.226346 24128 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.132:37847 I0220 18:44:25.283890 24130 ts_manager.cc:278] Set tserver state for 493be4abe83d46c1829e3d5cfc68a9de to MAINTENANCE_MODE I0220 18:44:25.284334 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 24165 W0220 18:44:25.290596 24555 connection.cc:658] client connection to 127.23.135.129:35559 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) W0220 18:44:25.290655 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: recv got EOF from 127.23.135.129:35559 (error 108). This is attempt 1: this message will repeat every 5th retry. W0220 18:44:25.290717 24555 meta_cache.cc:234] Tablet 827aa98f727b4da6a5015391b4da1960: Replica 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559) has failed: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) W0220 18:44:25.290982 24434 connection.cc:658] server connection from 127.23.135.129:50775 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) W0220 18:44:25.291105 24433 connection.cc:658] client connection to 127.23.135.129:35559 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) W0220 18:44:25.291183 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry. W0220 18:44:25.295101 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.296306 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.296362 24337 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.297539 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.297580 24337 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.297688 24336 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.298666 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.299481 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.301358 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.302057 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.302834 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.302855 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.313406 24336 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.322393 24336 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.322394 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.324458 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.329241 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.332499 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.333438 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.333438 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.337078 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.339432 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.339432 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.353780 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.356041 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.359395 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.363457 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.364830 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.364867 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.365871 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.368836 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.369689 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0220 18:44:25.395021 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.396080 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.401206 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.403334 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.404654 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.404672 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.410794 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.418437 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.421857 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.445896 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.446743 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.453065 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.464414 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.466444 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.469835 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.470701 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.475124 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.476171 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.507354 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.509300 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.511746 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.513717 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.516010 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.517313 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.530656 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.532857 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.533634 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.553956 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } I0220 18:44:25.572356 24571 raft_consensus.cc:482] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 493be4abe83d46c1829e3d5cfc68a9de) I0220 18:44:25.572475 24571 raft_consensus.cc:504] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:25.572768 24571 leader_election.cc:293] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:25.573216 24499 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "827aa98f727b4da6a5015391b4da1960" candidate_uuid: "c8a15b6a57804d05ada89b15de898a87" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" is_pre_election: true W0220 18:44:25.573240 24309 leader_election.cc:339] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111) I0220 18:44:25.573585 24308 leader_election.cc:307] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8a15b6a57804d05ada89b15de898a87; no voters: 493be4abe83d46c1829e3d5cfc68a9de, bccadedc7a11414384f1d4875bf49aaa I0220 18:44:25.573719 24571 raft_consensus.cc:2675] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0220 18:44:25.574298 24648 raft_consensus.cc:482] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 493be4abe83d46c1829e3d5cfc68a9de) I0220 18:44:25.574405 24648 raft_consensus.cc:504] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:25.574664 24648 leader_election.cc:293] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559), c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621) I0220 18:44:25.575064 24377 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "827aa98f727b4da6a5015391b4da1960" candidate_uuid: "bccadedc7a11414384f1d4875bf49aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" is_pre_election: true W0220 18:44:25.575088 24433 leader_election.cc:339] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111) I0220 18:44:25.575240 24648 raft_consensus.cc:482] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 493be4abe83d46c1829e3d5cfc68a9de) I0220 18:44:25.575242 24377 raft_consensus.cc:2414] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bccadedc7a11414384f1d4875bf49aaa in term 1. I0220 18:44:25.575340 24648 raft_consensus.cc:504] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:25.575525 24432 leader_election.cc:307] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87; no voters: 493be4abe83d46c1829e3d5cfc68a9de I0220 18:44:25.575613 24648 leader_election.cc:293] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621), 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559) I0220 18:44:25.575713 24646 raft_consensus.cc:2730] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Leader pre-election won for term 2 I0220 18:44:25.575748 24646 raft_consensus.cc:482] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 493be4abe83d46c1829e3d5cfc68a9de) I0220 18:44:25.575774 24646 raft_consensus.cc:2983] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Advancing to term 2 I0220 18:44:25.575920 24571 raft_consensus.cc:482] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 493be4abe83d46c1829e3d5cfc68a9de) I0220 18:44:25.576028 24571 raft_consensus.cc:504] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:25.576340 24571 leader_election.cc:293] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) W0220 18:44:25.576572 24433 leader_election.cc:339] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111) W0220 18:44:25.576639 24309 leader_election.cc:339] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111) I0220 18:44:25.576807 24646 raft_consensus.cc:504] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:25.577050 24646 leader_election.cc:293] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 2 election: Requested vote from peers 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559), c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621) I0220 18:44:25.577322 24377 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "2ca7a33c93d548f49d2a4b1eeca470ca" candidate_uuid: "bccadedc7a11414384f1d4875bf49aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" is_pre_election: true I0220 18:44:25.577394 24377 raft_consensus.cc:2414] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bccadedc7a11414384f1d4875bf49aaa in term 1. I0220 18:44:25.577391 24499 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "2ca7a33c93d548f49d2a4b1eeca470ca" candidate_uuid: "c8a15b6a57804d05ada89b15de898a87" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" is_pre_election: true I0220 18:44:25.577423 24375 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "827aa98f727b4da6a5015391b4da1960" candidate_uuid: "bccadedc7a11414384f1d4875bf49aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" I0220 18:44:25.577451 24499 raft_consensus.cc:2414] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a15b6a57804d05ada89b15de898a87 in term 1. I0220 18:44:25.577484 24375 raft_consensus.cc:2983] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Advancing to term 2 W0220 18:44:25.577674 24433 leader_election.cc:339] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111) I0220 18:44:25.577693 24308 leader_election.cc:307] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87; no voters: 493be4abe83d46c1829e3d5cfc68a9de I0220 18:44:25.577808 24571 raft_consensus.cc:2730] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0220 18:44:25.577831 24571 raft_consensus.cc:482] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 493be4abe83d46c1829e3d5cfc68a9de) I0220 18:44:25.577847 24571 raft_consensus.cc:2983] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Advancing to term 2 I0220 18:44:25.578171 24651 raft_consensus.cc:482] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 493be4abe83d46c1829e3d5cfc68a9de) I0220 18:44:25.578243 24651 raft_consensus.cc:504] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } I0220 18:44:25.578438 24375 raft_consensus.cc:2414] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bccadedc7a11414384f1d4875bf49aaa in term 2. I0220 18:44:25.578462 24651 leader_election.cc:293] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803), 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559) I0220 18:44:25.578696 24571 raft_consensus.cc:504] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:25.578707 24432 leader_election.cc:307] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87; no voters: 493be4abe83d46c1829e3d5cfc68a9de W0220 18:44:25.578884 24309 leader_election.cc:339] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111) I0220 18:44:25.578900 24571 leader_election.cc:293] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 2 election: Requested vote from peers 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:25.578981 24432 leader_election.cc:307] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87; no voters: 493be4abe83d46c1829e3d5cfc68a9de I0220 18:44:25.579062 24499 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "116ca55e12164875ad0a87183fe531bc" candidate_uuid: "c8a15b6a57804d05ada89b15de898a87" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" is_pre_election: true I0220 18:44:25.579159 24648 raft_consensus.cc:2730] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Leader pre-election won for term 2 I0220 18:44:25.579193 24648 raft_consensus.cc:482] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 493be4abe83d46c1829e3d5cfc68a9de) I0220 18:44:25.579219 24648 raft_consensus.cc:2983] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Advancing to term 2 W0220 18:44:25.579298 24309 leader_election.cc:339] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111) I0220 18:44:25.579584 24499 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "2ca7a33c93d548f49d2a4b1eeca470ca" candidate_uuid: "c8a15b6a57804d05ada89b15de898a87" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" I0220 18:44:25.579836 24308 leader_election.cc:307] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8a15b6a57804d05ada89b15de898a87; no voters: 493be4abe83d46c1829e3d5cfc68a9de, bccadedc7a11414384f1d4875bf49aaa I0220 18:44:25.580032 24648 raft_consensus.cc:504] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:25.580098 24571 raft_consensus.cc:2675] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0220 18:44:25.580174 24499 raft_consensus.cc:2339] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c8a15b6a57804d05ada89b15de898a87 in current term 2: Already voted for candidate bccadedc7a11414384f1d4875bf49aaa in this term. I0220 18:44:25.580269 24648 leader_election.cc:293] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 2 election: Requested vote from peers c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621), 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559) I0220 18:44:25.580334 24648 raft_consensus.cc:2730] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [term 2 FOLLOWER]: Leader election won for term 2 I0220 18:44:25.580390 24648 raft_consensus.cc:686] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [term 2 LEADER]: Becoming Leader. State: Replica: bccadedc7a11414384f1d4875bf49aaa, State: Running, Role: LEADER I0220 18:44:25.580513 24648 consensus_queue.cc:229] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } W0220 18:44:25.580828 24433 leader_election.cc:339] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111) I0220 18:44:25.581151 24308 leader_election.cc:307] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8a15b6a57804d05ada89b15de898a87; no voters: 493be4abe83d46c1829e3d5cfc68a9de, bccadedc7a11414384f1d4875bf49aaa I0220 18:44:25.581267 24571 raft_consensus.cc:2675] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority I0220 18:44:25.581282 24375 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "2ca7a33c93d548f49d2a4b1eeca470ca" candidate_uuid: "bccadedc7a11414384f1d4875bf49aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" I0220 18:44:25.581358 24375 raft_consensus.cc:2339] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bccadedc7a11414384f1d4875bf49aaa in current term 2: Already voted for candidate c8a15b6a57804d05ada89b15de898a87 in this term. I0220 18:44:25.581612 24432 leader_election.cc:307] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bccadedc7a11414384f1d4875bf49aaa; no voters: 493be4abe83d46c1829e3d5cfc68a9de, c8a15b6a57804d05ada89b15de898a87 I0220 18:44:25.581759 24646 raft_consensus.cc:2675] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority I0220 18:44:25.581807 24128 catalog_manager.cc:4173] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa reported cstate change: term changed from 1 to 2, leader changed from 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129) to bccadedc7a11414384f1d4875bf49aaa (127.23.135.131). New cstate: current_term: 2 leader_uuid: "bccadedc7a11414384f1d4875bf49aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } health_report { overall_health: HEALTHY } } } W0220 18:44:25.582566 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.582582 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } I0220 18:44:25.583284 24646 raft_consensus.cc:482] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 493be4abe83d46c1829e3d5cfc68a9de) I0220 18:44:25.583357 24646 raft_consensus.cc:504] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } I0220 18:44:25.583580 24646 leader_election.cc:293] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621), 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559) W0220 18:44:25.584033 24433 leader_election.cc:339] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111) I0220 18:44:25.584116 24375 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "116ca55e12164875ad0a87183fe531bc" candidate_uuid: "bccadedc7a11414384f1d4875bf49aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" is_pre_election: true I0220 18:44:25.584174 24375 raft_consensus.cc:2414] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bccadedc7a11414384f1d4875bf49aaa in term 1. W0220 18:44:25.584554 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "493be4abe83d46c1829e3d5cfc68a9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } } W0220 18:44:25.584837 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } I0220 18:44:25.585124 24432 leader_election.cc:307] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87; no voters: 493be4abe83d46c1829e3d5cfc68a9de I0220 18:44:25.585409 24646 raft_consensus.cc:2730] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Leader pre-election won for term 2 I0220 18:44:25.585439 24646 raft_consensus.cc:482] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 493be4abe83d46c1829e3d5cfc68a9de) I0220 18:44:25.585455 24646 raft_consensus.cc:2983] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Advancing to term 2 I0220 18:44:25.586299 24646 raft_consensus.cc:504] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } I0220 18:44:25.586522 24646 leader_election.cc:293] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 2 election: Requested vote from peers c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621), 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559) I0220 18:44:25.586591 24375 raft_consensus.cc:1210] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [term 2 FOLLOWER]: Refusing update from remote peer bccadedc7a11414384f1d4875bf49aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 2 index: 18. (index mismatch) W0220 18:44:25.586616 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0220 18:44:25.586899 24375 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "116ca55e12164875ad0a87183fe531bc" candidate_uuid: "bccadedc7a11414384f1d4875bf49aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" I0220 18:44:25.586946 24375 raft_consensus.cc:2983] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Advancing to term 2 W0220 18:44:25.586993 24433 leader_election.cc:339] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111) I0220 18:44:25.587532 24375 raft_consensus.cc:2414] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bccadedc7a11414384f1d4875bf49aaa in term 2. I0220 18:44:25.587730 24646 consensus_queue.cc:1005] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s I0220 18:44:25.587821 24432 leader_election.cc:307] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87; no voters: 493be4abe83d46c1829e3d5cfc68a9de I0220 18:44:25.587921 24646 raft_consensus.cc:2730] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [term 2 FOLLOWER]: Leader election won for term 2 I0220 18:44:25.587975 24646 raft_consensus.cc:686] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [term 2 LEADER]: Becoming Leader. State: Replica: bccadedc7a11414384f1d4875bf49aaa, State: Running, Role: LEADER I0220 18:44:25.588102 24646 consensus_queue.cc:229] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.16, Last appended by leader: 16, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } I0220 18:44:25.590978 24128 catalog_manager.cc:4173] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa reported cstate change: term changed from 1 to 2, leader changed from 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129) to bccadedc7a11414384f1d4875bf49aaa (127.23.135.131). New cstate: current_term: 2 leader_uuid: "bccadedc7a11414384f1d4875bf49aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } health_report { overall_health: UNKNOWN } } } W0220 18:44:25.606148 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0220 18:44:25.606276 24377 raft_consensus.cc:1210] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87 [term 2 FOLLOWER]: Refusing update from remote peer bccadedc7a11414384f1d4875bf49aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 2 index: 18. (index mismatch) I0220 18:44:25.606775 24648 consensus_queue.cc:1005] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.000s W0220 18:44:25.611479 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.616000 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.621496 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.633960 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.634784 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.643590 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.662456 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.662456 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.666409 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.671609 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.696595 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.698038 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.712496 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.719715 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.742345 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.748634 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.757208 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.757243 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0220 18:44:25.757230 24455 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.789371 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.793529 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.803010 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0220 18:44:25.807853 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.815178 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.853243 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0220 18:44:25.853329 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.859449 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.874040 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.880605 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.888995 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.928831 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.948225 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.955945 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:25.962364 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } I0220 18:44:25.969874 24647 raft_consensus.cc:482] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:44:25.970006 24647 raft_consensus.cc:504] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:25.970006 24571 raft_consensus.cc:482] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:44:25.970110 24571 raft_consensus.cc:504] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:25.970353 24647 leader_election.cc:293] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621), 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559) I0220 18:44:25.970477 24571 leader_election.cc:293] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:25.970770 24375 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "2ca7a33c93d548f49d2a4b1eeca470ca" candidate_uuid: "bccadedc7a11414384f1d4875bf49aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" is_pre_election: true I0220 18:44:25.970844 24375 raft_consensus.cc:2414] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bccadedc7a11414384f1d4875bf49aaa in term 2. W0220 18:44:25.970973 24309 leader_election.cc:339] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111) I0220 18:44:25.971107 24432 leader_election.cc:307] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87; no voters: I0220 18:44:25.971163 24499 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "2ca7a33c93d548f49d2a4b1eeca470ca" candidate_uuid: "c8a15b6a57804d05ada89b15de898a87" candidate_term: 3 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" is_pre_election: true W0220 18:44:25.971182 24433 leader_election.cc:339] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111) I0220 18:44:25.971251 24499 raft_consensus.cc:2414] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a15b6a57804d05ada89b15de898a87 in term 2. I0220 18:44:25.971288 24647 raft_consensus.cc:2730] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 2 FOLLOWER]: Leader pre-election won for term 3 I0220 18:44:25.971314 24647 raft_consensus.cc:482] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0220 18:44:25.971333 24647 raft_consensus.cc:2983] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 2 FOLLOWER]: Advancing to term 3 I0220 18:44:25.971536 24308 leader_election.cc:307] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87; no voters: 493be4abe83d46c1829e3d5cfc68a9de I0220 18:44:25.971750 24571 raft_consensus.cc:2730] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0220 18:44:25.971778 24571 raft_consensus.cc:482] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0220 18:44:25.971794 24571 raft_consensus.cc:2983] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 2 FOLLOWER]: Advancing to term 3 I0220 18:44:25.972250 24647 raft_consensus.cc:504] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:25.972467 24647 leader_election.cc:293] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 3 election: Requested vote from peers c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621), 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559) I0220 18:44:25.972515 24571 raft_consensus.cc:504] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:25.972772 24571 leader_election.cc:293] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 3 election: Requested vote from peers 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:25.972892 24375 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "2ca7a33c93d548f49d2a4b1eeca470ca" candidate_uuid: "bccadedc7a11414384f1d4875bf49aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" I0220 18:44:25.972954 24375 raft_consensus.cc:2339] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bccadedc7a11414384f1d4875bf49aaa in current term 3: Already voted for candidate c8a15b6a57804d05ada89b15de898a87 in this term. W0220 18:44:25.972998 24433 leader_election.cc:339] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111) W0220 18:44:25.973078 24309 leader_election.cc:339] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111) I0220 18:44:25.973104 24499 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "2ca7a33c93d548f49d2a4b1eeca470ca" candidate_uuid: "c8a15b6a57804d05ada89b15de898a87" candidate_term: 3 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" I0220 18:44:25.973162 24499 raft_consensus.cc:2339] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c8a15b6a57804d05ada89b15de898a87 in current term 3: Already voted for candidate bccadedc7a11414384f1d4875bf49aaa in this term. I0220 18:44:25.973356 24432 leader_election.cc:307] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bccadedc7a11414384f1d4875bf49aaa; no voters: 493be4abe83d46c1829e3d5cfc68a9de, c8a15b6a57804d05ada89b15de898a87 I0220 18:44:25.973484 24647 raft_consensus.cc:2675] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority I0220 18:44:25.973598 24308 leader_election.cc:307] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8a15b6a57804d05ada89b15de898a87; no voters: 493be4abe83d46c1829e3d5cfc68a9de, bccadedc7a11414384f1d4875bf49aaa I0220 18:44:25.973731 24571 raft_consensus.cc:2675] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority W0220 18:44:26.000758 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.008997 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.018569 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.037323 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.072652 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0220 18:44:26.076298 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.095484 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.100838 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.112428 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0220 18:44:26.114430 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.134351 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.136703 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.194430 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.203440 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.219915 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } I0220 18:44:26.228297 24772 heartbeater.cc:447] Master 127.23.135.190:45127 was elected leader, sending a full tablet report... W0220 18:44:26.235317 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.237455 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.264031 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.309329 24555 meta_cache.cc:234] Tablet 2ca7a33c93d548f49d2a4b1eeca470ca: Replica 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559) has failed: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111) W0220 18:44:26.309566 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0220 18:44:26.312038 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0220 18:44:26.314436 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.325923 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.347158 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0220 18:44:26.353421 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.354089 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.372448 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.399291 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.432500 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.468330 24335 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:54622: Illegal state: Replica c8a15b6a57804d05ada89b15de898a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.471748 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.498966 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } W0220 18:44:26.519013 24454 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:45212: Illegal state: Replica bccadedc7a11414384f1d4875bf49aaa is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } } I0220 18:44:26.540498 24571 raft_consensus.cc:482] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:44:26.540637 24571 raft_consensus.cc:504] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:26.540976 24571 leader_election.cc:293] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) W0220 18:44:26.541399 24309 leader_election.cc:339] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111) I0220 18:44:26.541416 24499 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "2ca7a33c93d548f49d2a4b1eeca470ca" candidate_uuid: "c8a15b6a57804d05ada89b15de898a87" candidate_term: 4 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" is_pre_election: true I0220 18:44:26.541505 24499 raft_consensus.cc:2414] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a15b6a57804d05ada89b15de898a87 in term 3. I0220 18:44:26.541781 24308 leader_election.cc:307] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87; no voters: 493be4abe83d46c1829e3d5cfc68a9de I0220 18:44:26.541930 24571 raft_consensus.cc:2730] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 3 FOLLOWER]: Leader pre-election won for term 4 I0220 18:44:26.541955 24571 raft_consensus.cc:482] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0220 18:44:26.541975 24571 raft_consensus.cc:2983] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 3 FOLLOWER]: Advancing to term 4 I0220 18:44:26.542685 24571 raft_consensus.cc:504] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:26.542908 24571 leader_election.cc:293] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 4 election: Requested vote from peers 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) W0220 18:44:26.543244 24309 leader_election.cc:339] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111) I0220 18:44:26.543283 24499 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "2ca7a33c93d548f49d2a4b1eeca470ca" candidate_uuid: "c8a15b6a57804d05ada89b15de898a87" candidate_term: 4 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" I0220 18:44:26.543339 24499 raft_consensus.cc:2983] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 3 FOLLOWER]: Advancing to term 4 I0220 18:44:26.544040 24499 raft_consensus.cc:2414] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a15b6a57804d05ada89b15de898a87 in term 4. I0220 18:44:26.544297 24308 leader_election.cc:307] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87; no voters: 493be4abe83d46c1829e3d5cfc68a9de I0220 18:44:26.544431 24571 raft_consensus.cc:2730] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 4 FOLLOWER]: Leader election won for term 4 I0220 18:44:26.544560 24571 raft_consensus.cc:686] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 4 LEADER]: Becoming Leader. State: Replica: c8a15b6a57804d05ada89b15de898a87, State: Running, Role: LEADER I0220 18:44:26.544694 24571 consensus_queue.cc:229] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.16, Last appended by leader: 16, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:26.546161 24128 catalog_manager.cc:4173] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 reported cstate change: term changed from 1 to 4, leader changed from 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129) to c8a15b6a57804d05ada89b15de898a87 (127.23.135.130). New cstate: current_term: 4 leader_uuid: "c8a15b6a57804d05ada89b15de898a87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } health_report { overall_health: UNKNOWN } } } W0220 18:44:26.549860 24309 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0220 18:44:26.549933 24499 raft_consensus.cc:1210] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 4 FOLLOWER]: Refusing update from remote peer c8a15b6a57804d05ada89b15de898a87: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 4 index: 18. (index mismatch) I0220 18:44:26.550346 24776 consensus_queue.cc:1005] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s I0220 18:44:26.553898 24784 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:54622 (ReqId={client: 392f3f16a48c4ca3af0be6adf713ed44, seq_no=87, attempt_no=1}) took 1257ms. Trace: I0220 18:44:26.553936 24784 rpcz_store.cc:270] 0220 18:44:25.296021 (+ 0us) service_pool.cc:162] Inserting onto call queue 0220 18:44:25.296064 (+ 43us) service_pool.cc:221] Handling call 0220 18:44:26.553855 (+1257791us) inbound_call.cc:162] Queueing success response Metrics: {} W0220 18:44:26.583773 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0220 18:44:26.667307 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0220 18:44:26.796339 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0220 18:44:26.810386 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0220 18:44:26.820940 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0220 18:44:27.050640 24309 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0220 18:44:27.098259 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0220 18:44:27.151875 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. I0220 18:44:27.284153 24796 consensus_queue.cc:565] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Leader has been unable to successfully communicate with peer 493be4abe83d46c1829e3d5cfc68a9de for more than 2 seconds (2.001s) W0220 18:44:27.286803 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. I0220 18:44:27.293329 24807 consensus_queue.cc:565] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [LEADER]: Leader has been unable to successfully communicate with peer 493be4abe83d46c1829e3d5cfc68a9de for more than 2 seconds (2.011s) W0220 18:44:27.294840 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. I0220 18:44:27.298823 24795 consensus_queue.cc:565] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Leader has been unable to successfully communicate with peer 493be4abe83d46c1829e3d5cfc68a9de for more than 2 seconds (2.035s) W0220 18:44:27.301268 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. W0220 18:44:27.551307 24309 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. I0220 18:44:27.612128 24795 consensus_queue.cc:565] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Leader has been unable to successfully communicate with peer 493be4abe83d46c1829e3d5cfc68a9de for more than 2 seconds (2.024s) I0220 18:44:27.617236 24796 consensus_queue.cc:565] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Leader has been unable to successfully communicate with peer 493be4abe83d46c1829e3d5cfc68a9de for more than 2 seconds (2.037s) W0220 18:44:27.621660 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. W0220 18:44:27.739526 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. W0220 18:44:27.790971 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. W0220 18:44:27.814695 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. W0220 18:44:27.862412 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. W0220 18:44:28.067376 24309 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0220 18:44:28.107436 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. W0220 18:44:28.227804 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. W0220 18:44:28.323710 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. W0220 18:44:28.351420 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. W0220 18:44:28.376207 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. I0220 18:44:28.586123 24786 consensus_queue.cc:565] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [LEADER]: Leader has been unable to successfully communicate with peer 493be4abe83d46c1829e3d5cfc68a9de for more than 2 seconds (2.041s) W0220 18:44:28.590602 24309 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. W0220 18:44:28.598731 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. I0220 18:44:28.676458 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-master with pid 24102 I0220 18:44:28.684304 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-master /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.23.135.190:45127 --webserver_interface=127.23.135.190 --webserver_port=38291 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:41009 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --rpc_reuseport=true with env {} W0220 18:44:28.772948 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:44:28.774454 24819 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:44:28.774715 24819 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:44:28.776290 24819 flags.cc:403] Enabled experimental flag: --ipki_ca_key_size=1024 W0220 18:44:28.776402 24819 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:44:28.776468 24819 flags.cc:403] Enabled experimental flag: --tsk_num_rsa_bits=512 W0220 18:44:28.776525 24819 flags.cc:403] Enabled experimental flag: --rpc_reuseport=true I0220 18:44:28.779105 24819 master_runner.cc:90] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:41009 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.23.135.190:45127 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/data/info.pb --webserver_interface=127.23.135.190 --webserver_port=38291 --never_fsync=true --heap_profile_path=/tmp/kudu-master.24819 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-master --enable_minidumps=false --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:44:28.779901 24819 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:44:28.781204 24819 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:44:28.788009 24819 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:44:28.789309 24819 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:44:28.796699 24819 hybrid_clock.cc:552] HybridClock initialized: now 1582224268796602 us; error 83 us; skew 500 ppm I0220 18:44:28.797691 24819 fs_manager.cc:259] Metadata directory not provided I0220 18:44:28.797819 24819 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory W0220 18:44:28.805589 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry. I0220 18:44:28.833727 24819 fs_manager.cc:410] Time spent opening directory manager: real 0.035s user 0.007s sys 0.000s I0220 18:44:28.844539 24819 fs_manager.cc:430] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s I0220 18:44:28.846748 24819 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/wal uuid: "1b8314706b1247e7a519c16d52957797" format_stamp: "Formatted at 2020-02-20 18:44:24 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:28.847805 24819 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:44:28.861413 24827 reactor.cc:496] Calling ReactorThread::RunThread()... W0220 18:44:28.877388 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry. W0220 18:44:28.906489 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry. I0220 18:44:28.924638 24819 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:44:28.938321 24819 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.190:45127 I0220 18:44:28.943949 24819 webserver.cc:382] Webserver started at http://127.23.135.190:38291/ using document root and password file I0220 18:44:28.944742 24819 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/master-0/data/info.pb I0220 18:44:28.947841 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-master as pid 24819 I0220 18:44:28.949564 24880 tablet_bootstrap.cc:441] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797: Bootstrap starting. I0220 18:44:28.964274 24880 log.cc:828] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797: Log is configured to *not* fsync() on all Append() calls I0220 18:44:28.988085 24880 tablet_bootstrap.cc:441] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=10 ignored=0} mutations{seen=15 ignored=0} orphaned_commits=0. Pending: 0 replicates I0220 18:44:28.988620 24880 tablet_bootstrap.cc:441] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797: Bootstrap complete. I0220 18:44:29.004179 24880 raft_consensus.cc:346] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8314706b1247e7a519c16d52957797" member_type: VOTER } I0220 18:44:29.004441 24880 raft_consensus.cc:726] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b8314706b1247e7a519c16d52957797, State: Initialized, Role: FOLLOWER I0220 18:44:29.005038 24880 consensus_queue.cc:252] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8314706b1247e7a519c16d52957797" member_type: VOTER } I0220 18:44:29.005362 24880 raft_consensus.cc:386] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0220 18:44:29.005448 24880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0220 18:44:29.005568 24880 raft_consensus.cc:2983] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [term 1 FOLLOWER]: Advancing to term 2 I0220 18:44:29.007750 24880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8314706b1247e7a519c16d52957797" member_type: VOTER } I0220 18:44:29.008257 24880 leader_election.cc:307] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b8314706b1247e7a519c16d52957797; no voters: I0220 18:44:29.008821 24880 leader_election.cc:293] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [CANDIDATE]: Term 2 election: Requested vote from peers I0220 18:44:29.009208 24880 sys_catalog.cc:365] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8314706b1247e7a519c16d52957797" member_type: VOTER } } I0220 18:44:29.009311 24880 sys_catalog.cc:368] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [sys.catalog]: This master's current role is: FOLLOWER I0220 18:44:29.010077 24880 sys_catalog.cc:466] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [sys.catalog]: configured and running, proceeding with master startup. I0220 18:44:29.010313 24882 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [term 2 FOLLOWER]: Leader election won for term 2 I0220 18:44:29.010483 24882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [term 2 LEADER]: Becoming Leader. State: Replica: 1b8314706b1247e7a519c16d52957797, State: Running, Role: LEADER I0220 18:44:29.010785 24882 consensus_queue.cc:229] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8314706b1247e7a519c16d52957797" member_type: VOTER } I0220 18:44:29.012302 24882 sys_catalog.cc:365] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b8314706b1247e7a519c16d52957797. Latest consensus state: current_term: 2 leader_uuid: "1b8314706b1247e7a519c16d52957797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8314706b1247e7a519c16d52957797" member_type: VOTER } } I0220 18:44:29.014847 24882 sys_catalog.cc:368] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [sys.catalog]: This master's current role is: LEADER I0220 18:44:29.012738 24883 sys_catalog.cc:365] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1b8314706b1247e7a519c16d52957797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8314706b1247e7a519c16d52957797" member_type: VOTER } } I0220 18:44:29.015164 24883 sys_catalog.cc:368] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797 [sys.catalog]: This master's current role is: LEADER I0220 18:44:29.017948 24884 catalog_manager.cc:1174] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797: acquiring CA information for follower catalog manager: success I0220 18:44:29.019563 24884 catalog_manager.cc:1202] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0220 18:44:29.019675 24888 catalog_manager.cc:1107] Loading table and tablet metadata into memory... I0220 18:44:29.021073 24888 catalog_manager.cc:408] Loaded metadata for table test-workload [id=73578aeb8f9e4f9597b79f5737a60111] I0220 18:44:29.022157 24888 catalog_manager.cc:462] Loaded metadata for tablet 116ca55e12164875ad0a87183fe531bc (table test-workload [id=73578aeb8f9e4f9597b79f5737a60111]) I0220 18:44:29.022620 24888 catalog_manager.cc:462] Loaded metadata for tablet 2ca7a33c93d548f49d2a4b1eeca470ca (table test-workload [id=73578aeb8f9e4f9597b79f5737a60111]) I0220 18:44:29.022789 24888 catalog_manager.cc:462] Loaded metadata for tablet 827aa98f727b4da6a5015391b4da1960 (table test-workload [id=73578aeb8f9e4f9597b79f5737a60111]) I0220 18:44:29.022951 24888 catalog_manager.cc:462] Loaded metadata for tablet a7a069dabd5b4df28954b6833f23ab31 (table test-workload [id=73578aeb8f9e4f9597b79f5737a60111]) I0220 18:44:29.023097 24888 catalog_manager.cc:462] Loaded metadata for tablet c0ca382ac98b46948b2b18c2d691a364 (table test-workload [id=73578aeb8f9e4f9597b79f5737a60111]) I0220 18:44:29.023275 24888 catalog_manager.cc:462] Loaded metadata for tablet c5b2eb9c654f49df828ad8a096c62921 (table test-workload [id=73578aeb8f9e4f9597b79f5737a60111]) I0220 18:44:29.023356 24888 catalog_manager.cc:1118] Initializing Kudu internal certificate authority... I0220 18:44:29.024211 24888 catalog_manager.cc:1127] Loading token signing keys... I0220 18:44:29.024853 24888 catalog_manager.cc:4520] T 00000000000000000000000000000000 P 1b8314706b1247e7a519c16d52957797: Loaded TSK: 0 I0220 18:44:29.025418 24888 catalog_manager.cc:1137] Initializing in-progress tserver states... W0220 18:44:29.094063 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry. W0220 18:44:29.146634 24309 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. I0220 18:44:29.243746 24846 master_service.cc:296] Got heartbeat from unknown tserver (permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" instance_seqno: 1582224265208249) as {username='slave'} at 127.23.135.132:48341; Asking this server to re-register. I0220 18:44:29.244881 24772 heartbeater.cc:406] Registering TS with master... I0220 18:44:29.245004 24772 heartbeater.cc:455] Master 127.23.135.190:45127 requested a full tablet report, sending... I0220 18:44:29.246129 24846 ts_manager.cc:192] Registered new tserver with Master: 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132:45837) W0220 18:44:29.283159 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry. W0220 18:44:29.296277 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry. W0220 18:44:29.359549 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry. W0220 18:44:29.440798 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry. I0220 18:44:29.594583 24846 master_service.cc:296] Got heartbeat from unknown tserver (permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" instance_seqno: 1582224264679278) as {username='slave'} at 127.23.135.130:36223; Asking this server to re-register. I0220 18:44:29.595196 24420 heartbeater.cc:406] Registering TS with master... I0220 18:44:29.595293 24420 heartbeater.cc:455] Master 127.23.135.190:45127 requested a full tablet report, sending... I0220 18:44:29.596387 24846 ts_manager.cc:192] Registered new tserver with Master: c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621) I0220 18:44:29.626696 24846 master_service.cc:296] Got heartbeat from unknown tserver (permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" instance_seqno: 1582224264783299) as {username='slave'} at 127.23.135.131:44515; Asking this server to re-register. I0220 18:44:29.627168 24544 heartbeater.cc:406] Registering TS with master... I0220 18:44:29.627277 24544 heartbeater.cc:455] Master 127.23.135.190:45127 requested a full tablet report, sending... I0220 18:44:29.628271 24846 ts_manager.cc:192] Registered new tserver with Master: bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) W0220 18:44:29.641538 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry. W0220 18:44:29.664436 24309 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. W0220 18:44:29.807526 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry. W0220 18:44:29.825513 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry. W0220 18:44:29.909018 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry. W0220 18:44:29.936578 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry. W0220 18:44:30.145784 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry. W0220 18:44:30.151504 24309 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry. W0220 18:44:30.291378 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry. W0220 18:44:30.365767 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry. W0220 18:44:30.433513 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry. W0220 18:44:30.437088 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry. W0220 18:44:30.634651 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry. W0220 18:44:30.686007 24309 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry. W0220 18:44:30.830899 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry. W0220 18:44:30.923017 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry. W0220 18:44:30.967667 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry. W0220 18:44:31.012115 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry. W0220 18:44:31.158686 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry. W0220 18:44:31.240242 24309 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry. W0220 18:44:31.361838 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry. W0220 18:44:31.465348 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry. W0220 18:44:31.542157 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry. W0220 18:44:31.612354 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry. W0220 18:44:31.651751 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry. W0220 18:44:31.868160 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry. W0220 18:44:31.868495 24309 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry. I0220 18:44:31.976627 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.23.135.129:35559 --local_ip_for_outbound_sockets=127.23.135.129 --webserver_port=45061 --webserver_interface=127.23.135.129 --tserver_master_addrs=127.23.135.190:45127 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:41009 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --follower_unavailable_considered_failed_sec=2 --enable_log_gc=false with env {} W0220 18:44:32.004093 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry. WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:44:32.065403 24899 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:44:32.065656 24899 flags.cc:403] Enabled unsafe flag: --enable_log_gc=false W0220 18:44:32.065711 24899 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:44:32.066946 24899 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:44:32.067028 24899 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.129 I0220 18:44:32.069119 24899 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:41009 --ntp_initial_sync_wait_secs=10 --time_source=builtin --follower_unavailable_considered_failed_sec=2 --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.129:35559 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/data/info.pb --webserver_interface=127.23.135.129 --webserver_port=45061 --enable_log_gc=false --tserver_master_addrs=127.23.135.190:45127 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.24899 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.129 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:44:32.069705 24899 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:44:32.070535 24899 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:44:32.078851 24899 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:44:32.080291 24899 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:44:32.081784 24899 hybrid_clock.cc:552] HybridClock initialized: now 1582224272081685 us; error 92 us; skew 500 ppm I0220 18:44:32.082456 24899 fs_manager.cc:259] Metadata directory not provided I0220 18:44:32.082523 24899 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory W0220 18:44:32.110680 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry. W0220 18:44:32.113605 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry. I0220 18:44:32.117552 24899 fs_manager.cc:410] Time spent opening directory manager: real 0.032s user 0.006s sys 0.000s I0220 18:44:32.128114 24899 fs_manager.cc:430] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s I0220 18:44:32.128794 24899 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/wal uuid: "493be4abe83d46c1829e3d5cfc68a9de" format_stamp: "Formatted at 2020-02-20 18:44:24 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:32.129119 24899 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:44:32.181311 24899 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:44:32.183084 24899 ts_tablet_manager.cc:383] Loading tablet metadata (0/6 complete) I0220 18:44:32.189524 24899 ts_tablet_manager.cc:395] Loaded tablet metadata (6 total tablets, 6 live tablets) I0220 18:44:32.189649 24899 ts_tablet_manager.cc:401] Registering tablets (0/6 complete) I0220 18:44:32.192009 24916 tablet_bootstrap.cc:441] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap starting. I0220 18:44:32.197342 24899 ts_tablet_manager.cc:415] Registered 6 tablets I0220 18:44:32.210049 24899 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.129:35559 I0220 18:44:32.215179 24899 webserver.cc:382] Webserver started at http://127.23.135.129:45061/ using document root and password file I0220 18:44:32.215943 24899 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1582224264261468-24094/minicluster-data/ts-0/data/info.pb I0220 18:44:32.216823 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 24899 I0220 18:44:32.227834 24916 log.cc:828] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de: Log is configured to *not* fsync() on all Append() calls I0220 18:44:32.235584 25021 heartbeater.cc:322] Connected to a master server at 127.23.135.190:45127 I0220 18:44:32.235842 25021 heartbeater.cc:406] Registering TS with master... I0220 18:44:32.236353 25021 heartbeater.cc:455] Master 127.23.135.190:45127 requested a full tablet report, sending... I0220 18:44:32.238080 24846 ts_manager.cc:192] Registered new tserver with Master: 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559) I0220 18:44:32.240357 24846 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.129:48655 W0220 18:44:32.275465 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 66: this message will repeat every 5th retry. I0220 18:44:32.349565 24916 tablet_bootstrap.cc:441] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=17 ignored=0} inserts{seen=124 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0220 18:44:32.350168 24916 tablet_bootstrap.cc:441] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap complete. I0220 18:44:32.351289 24916 ts_tablet_manager.cc:1113] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de: Time spent bootstrapping tablet: real 0.159s user 0.039s sys 0.000s I0220 18:44:32.354754 24916 raft_consensus.cc:346] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:32.356245 24916 raft_consensus.cc:726] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493be4abe83d46c1829e3d5cfc68a9de, State: Initialized, Role: FOLLOWER I0220 18:44:32.356791 24916 consensus_queue.cc:252] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 1.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:32.357304 25021 heartbeater.cc:447] Master 127.23.135.190:45127 was elected leader, sending a full tablet report... I0220 18:44:32.359993 24916 ts_tablet_manager.cc:1142] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s I0220 18:44:32.360265 24916 tablet_bootstrap.cc:441] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap starting. I0220 18:44:32.402170 24916 tablet_bootstrap.cc:441] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=0 applied=16 ignored=0} inserts{seen=130 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0220 18:44:32.402477 24916 tablet_bootstrap.cc:441] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap complete. I0220 18:44:32.405004 24916 ts_tablet_manager.cc:1113] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de: Time spent bootstrapping tablet: real 0.045s user 0.018s sys 0.008s I0220 18:44:32.405808 24916 raft_consensus.cc:346] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:32.406280 24916 raft_consensus.cc:726] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493be4abe83d46c1829e3d5cfc68a9de, State: Initialized, Role: FOLLOWER I0220 18:44:32.406812 24916 consensus_queue.cc:252] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 1.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:32.407550 24916 ts_tablet_manager.cc:1142] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0220 18:44:32.407793 24916 tablet_bootstrap.cc:441] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap starting. I0220 18:44:32.434604 24916 tablet_bootstrap.cc:441] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=0 applied=16 ignored=0} inserts{seen=120 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0220 18:44:32.435042 24916 tablet_bootstrap.cc:441] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap complete. I0220 18:44:32.436026 24916 ts_tablet_manager.cc:1113] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.012s W0220 18:44:32.439952 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 66: this message will repeat every 5th retry. I0220 18:44:32.440299 24916 raft_consensus.cc:346] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:32.443691 24916 raft_consensus.cc:726] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493be4abe83d46c1829e3d5cfc68a9de, State: Initialized, Role: FOLLOWER I0220 18:44:32.443940 24916 consensus_queue.cc:252] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 1.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:32.444499 24916 ts_tablet_manager.cc:1142] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s I0220 18:44:32.444664 24916 tablet_bootstrap.cc:441] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap starting. W0220 18:44:32.469064 25022 tablet_replica_mm_ops.cc:302] Log GC is disabled (check --enable_log_gc) I0220 18:44:32.508666 24916 tablet_bootstrap.cc:441] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=0 applied=17 ignored=0} inserts{seen=139 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0220 18:44:32.509502 24916 tablet_bootstrap.cc:441] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap complete. I0220 18:44:32.515470 24916 ts_tablet_manager.cc:1113] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de: Time spent bootstrapping tablet: real 0.071s user 0.016s sys 0.011s I0220 18:44:32.517143 24916 raft_consensus.cc:346] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:32.520157 24916 raft_consensus.cc:726] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493be4abe83d46c1829e3d5cfc68a9de, State: Initialized, Role: FOLLOWER I0220 18:44:32.520584 24916 consensus_queue.cc:252] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 1.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:32.521196 24916 ts_tablet_manager.cc:1142] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s I0220 18:44:32.521847 24916 tablet_bootstrap.cc:441] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap starting. I0220 18:44:32.521347 24963 raft_consensus.cc:2983] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Advancing to term 4 I0220 18:44:32.525331 24963 pending_rounds.cc:78] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de: Aborting all transactions after (but not including) 16 I0220 18:44:32.525430 24963 pending_rounds.cc:99] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de: Aborting uncommitted WRITE_OP operation due to leader change: 1.17 I0220 18:44:32.544535 24970 raft_consensus.cc:2983] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Advancing to term 2 I0220 18:44:32.569839 24970 pending_rounds.cc:78] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de: Aborting all transactions after (but not including) 16 I0220 18:44:32.569877 24970 pending_rounds.cc:99] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de: Aborting uncommitted WRITE_OP operation due to leader change: 1.17 I0220 18:44:32.747225 24916 tablet_bootstrap.cc:441] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=141 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0220 18:44:32.747675 24916 tablet_bootstrap.cc:441] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap complete. I0220 18:44:32.748592 24916 ts_tablet_manager.cc:1113] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de: Time spent bootstrapping tablet: real 0.227s user 0.026s sys 0.004s I0220 18:44:32.749119 24916 raft_consensus.cc:346] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:32.749220 24916 raft_consensus.cc:726] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493be4abe83d46c1829e3d5cfc68a9de, State: Initialized, Role: FOLLOWER I0220 18:44:32.749438 24916 consensus_queue.cc:252] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 1.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:32.749785 24916 ts_tablet_manager.cc:1142] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I0220 18:44:32.750010 24916 tablet_bootstrap.cc:441] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap starting. I0220 18:44:32.834273 24916 tablet_bootstrap.cc:441] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=0 applied=14 ignored=0} inserts{seen=116 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates I0220 18:44:32.834753 24916 tablet_bootstrap.cc:441] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de: Bootstrap complete. I0220 18:44:32.860826 24916 ts_tablet_manager.cc:1113] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de: Time spent bootstrapping tablet: real 0.111s user 0.020s sys 0.007s I0220 18:44:32.861443 24916 raft_consensus.cc:346] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Replica starting. Triggering 3 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } I0220 18:44:32.861992 24916 raft_consensus.cc:726] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493be4abe83d46c1829e3d5cfc68a9de, State: Initialized, Role: FOLLOWER I0220 18:44:32.866320 24916 consensus_queue.cc:252] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 1.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } I0220 18:44:32.866801 24916 ts_tablet_manager.cc:1142] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s I0220 18:44:32.971710 25030 raft_consensus.cc:482] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c8a15b6a57804d05ada89b15de898a87) I0220 18:44:32.971861 25030 raft_consensus.cc:504] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:32.972940 25030 leader_election.cc:293] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:32.997673 24499 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "c5b2eb9c654f49df828ad8a096c62921" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 2 candidate_status { last_received { term: 1 index: 660 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" is_pre_election: true I0220 18:44:33.015035 24377 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "c5b2eb9c654f49df828ad8a096c62921" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 2 candidate_status { last_received { term: 1 index: 660 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" is_pre_election: true I0220 18:44:33.015578 24908 leader_election.cc:307] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 493be4abe83d46c1829e3d5cfc68a9de; no voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87 I0220 18:44:33.016156 25030 raft_consensus.cc:2675] T c5b2eb9c654f49df828ad8a096c62921 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0220 18:44:33.073586 24978 raft_consensus.cc:2983] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Advancing to term 2 I0220 18:44:33.075676 24978 pending_rounds.cc:78] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de: Aborting all transactions after (but not including) 16 I0220 18:44:33.075824 24978 pending_rounds.cc:99] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de: Aborting uncommitted WRITE_OP operation due to leader change: 1.17 I0220 18:44:33.189476 25047 raft_consensus.cc:482] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bccadedc7a11414384f1d4875bf49aaa) I0220 18:44:33.189611 25047 raft_consensus.cc:504] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:33.189924 25047 leader_election.cc:293] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:33.190363 24376 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "827aa98f727b4da6a5015391b4da1960" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 3 candidate_status { last_received { term: 2 index: 663 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" is_pre_election: true I0220 18:44:33.191856 24499 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "827aa98f727b4da6a5015391b4da1960" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 3 candidate_status { last_received { term: 2 index: 663 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" is_pre_election: true I0220 18:44:33.192842 24908 leader_election.cc:307] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 493be4abe83d46c1829e3d5cfc68a9de; no voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87 I0220 18:44:33.206164 25047 raft_consensus.cc:2675] T 827aa98f727b4da6a5015391b4da1960 P 493be4abe83d46c1829e3d5cfc68a9de [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0220 18:44:33.259348 25046 raft_consensus.cc:482] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bccadedc7a11414384f1d4875bf49aaa) I0220 18:44:33.259476 25046 raft_consensus.cc:504] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:33.259987 25046 leader_election.cc:293] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:33.260583 24377 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "c0ca382ac98b46948b2b18c2d691a364" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 2 candidate_status { last_received { term: 1 index: 671 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" is_pre_election: true I0220 18:44:33.261224 24501 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "c0ca382ac98b46948b2b18c2d691a364" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 2 candidate_status { last_received { term: 1 index: 671 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" is_pre_election: true I0220 18:44:33.261593 24908 leader_election.cc:307] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 493be4abe83d46c1829e3d5cfc68a9de; no voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87 I0220 18:44:33.261817 25046 raft_consensus.cc:2675] T c0ca382ac98b46948b2b18c2d691a364 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0220 18:44:33.275411 25040 raft_consensus.cc:482] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c8a15b6a57804d05ada89b15de898a87) I0220 18:44:33.275537 25040 raft_consensus.cc:504] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:33.276026 25040 leader_election.cc:293] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:33.276643 24377 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "2ca7a33c93d548f49d2a4b1eeca470ca" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 5 candidate_status { last_received { term: 4 index: 663 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" is_pre_election: true I0220 18:44:33.278452 24501 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "2ca7a33c93d548f49d2a4b1eeca470ca" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 5 candidate_status { last_received { term: 4 index: 663 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" is_pre_election: true I0220 18:44:33.278762 24908 leader_election.cc:307] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 493be4abe83d46c1829e3d5cfc68a9de; no voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87 I0220 18:44:33.278889 25040 raft_consensus.cc:2675] T 2ca7a33c93d548f49d2a4b1eeca470ca P 493be4abe83d46c1829e3d5cfc68a9de [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0220 18:44:33.444903 25047 raft_consensus.cc:482] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bccadedc7a11414384f1d4875bf49aaa) I0220 18:44:33.445022 25047 raft_consensus.cc:504] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } I0220 18:44:33.445340 25047 leader_election.cc:293] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:33.446203 24376 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "a7a069dabd5b4df28954b6833f23ab31" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 2 candidate_status { last_received { term: 1 index: 682 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" is_pre_election: true I0220 18:44:33.462201 24501 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "a7a069dabd5b4df28954b6833f23ab31" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 2 candidate_status { last_received { term: 1 index: 682 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" is_pre_election: true I0220 18:44:33.462690 24908 leader_election.cc:307] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 493be4abe83d46c1829e3d5cfc68a9de; no voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87 I0220 18:44:33.462894 25047 raft_consensus.cc:2675] T a7a069dabd5b4df28954b6833f23ab31 P 493be4abe83d46c1829e3d5cfc68a9de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0220 18:44:33.464190 25040 raft_consensus.cc:482] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bccadedc7a11414384f1d4875bf49aaa) I0220 18:44:33.464285 25040 raft_consensus.cc:504] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } I0220 18:44:33.464576 25040 leader_election.cc:293] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621), bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:33.465031 24499 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "116ca55e12164875ad0a87183fe531bc" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 3 candidate_status { last_received { term: 2 index: 692 } } ignore_live_leader: false dest_uuid: "bccadedc7a11414384f1d4875bf49aaa" is_pre_election: true I0220 18:44:33.465867 24377 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "116ca55e12164875ad0a87183fe531bc" candidate_uuid: "493be4abe83d46c1829e3d5cfc68a9de" candidate_term: 3 candidate_status { last_received { term: 2 index: 692 } } ignore_live_leader: false dest_uuid: "c8a15b6a57804d05ada89b15de898a87" is_pre_election: true I0220 18:44:33.466181 24908 leader_election.cc:307] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 493be4abe83d46c1829e3d5cfc68a9de; no voters: bccadedc7a11414384f1d4875bf49aaa, c8a15b6a57804d05ada89b15de898a87 I0220 18:44:33.466423 25040 raft_consensus.cc:2675] T 116ca55e12164875ad0a87183fe531bc P 493be4abe83d46c1829e3d5cfc68a9de [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0220 18:44:33.693147 24842 ts_manager.cc:267] Unset tserver state for 493be4abe83d46c1829e3d5cfc68a9de from MAINTENANCE_MODE I0220 18:44:33.912380 25037 mvcc.cc:203] Tried to move safe_time back from 6480790619718496256 to 6480790595767803904. Current Snapshot: MvccSnapshot[committed={T|T < 6480790600096358400 or (T in {6480790600098631680,6480790600130088960,6480790600229715968,6480790600234577920,6480790600242290688,6480790600273170432,6480790600344485888,6480790600361533440,6480790600370593792,6480790600413396992,6480790600496951296,6480790600503562240,6480790600511885312})}] I0220 18:44:34.255997 24772 heartbeater.cc:455] Master 127.23.135.190:45127 requested a full tablet report, sending... I0220 18:44:34.448652 24420 heartbeater.cc:455] Master 127.23.135.190:45127 requested a full tablet report, sending... I0220 18:44:34.468983 25021 heartbeater.cc:455] Master 127.23.135.190:45127 requested a full tablet report, sending... I0220 18:44:34.609895 24544 heartbeater.cc:455] Master 127.23.135.190:45127 requested a full tablet report, sending... W0220 18:44:35.705600 24432 outbound_call.cc:284] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.135.130:40621, user_credentials={real_user=slave}} blocked reactor thread for 43690.3us I0220 18:44:36.739976 24842 ts_manager.cc:278] Set tserver state for 493be4abe83d46c1829e3d5cfc68a9de to MAINTENANCE_MODE I0220 18:44:36.740427 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 24899 W0220 18:44:36.747615 24433 connection.cc:658] client connection to 127.23.135.129:35559 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) W0220 18:44:36.747808 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry. W0220 18:44:36.748113 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry. W0220 18:44:36.748201 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry. W0220 18:44:36.748263 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry. W0220 18:44:36.762967 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0220 18:44:36.763128 24309 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0220 18:44:37.225502 24309 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0220 18:44:37.229194 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0220 18:44:37.229334 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0220 18:44:37.237424 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0220 18:44:37.256914 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0220 18:44:37.291863 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0220 18:44:37.740689 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0220 18:44:37.741590 24309 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0220 18:44:37.748404 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0220 18:44:37.750229 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0220 18:44:37.753674 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0220 18:44:37.794319 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0220 18:44:38.216835 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0220 18:44:38.229266 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0220 18:44:38.261399 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0220 18:44:38.275169 24309 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0220 18:44:38.303802 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0220 18:44:38.359947 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0220 18:44:38.689234 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. W0220 18:44:38.727694 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. I0220 18:44:38.763041 25074 consensus_queue.cc:565] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Leader has been unable to successfully communicate with peer 493be4abe83d46c1829e3d5cfc68a9de for more than 2 seconds (2.029s) I0220 18:44:38.769909 25059 consensus_queue.cc:565] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [LEADER]: Leader has been unable to successfully communicate with peer 493be4abe83d46c1829e3d5cfc68a9de for more than 2 seconds (2.032s) W0220 18:44:38.774386 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. I0220 18:44:38.775264 25074 consensus_queue.cc:565] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Leader has been unable to successfully communicate with peer 493be4abe83d46c1829e3d5cfc68a9de for more than 2 seconds (2.029s) I0220 18:44:38.775461 25082 consensus_queue.cc:565] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Leader has been unable to successfully communicate with peer 493be4abe83d46c1829e3d5cfc68a9de for more than 2 seconds (2.049s) I0220 18:44:38.790735 25099 consensus_queue.cc:565] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [LEADER]: Leader has been unable to successfully communicate with peer 493be4abe83d46c1829e3d5cfc68a9de for more than 2 seconds (2.055s) W0220 18:44:38.793882 24309 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. W0220 18:44:38.794206 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. I0220 18:44:38.823190 25082 consensus_queue.cc:565] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Leader has been unable to successfully communicate with peer 493be4abe83d46c1829e3d5cfc68a9de for more than 2 seconds (2.083s) W0220 18:44:38.875188 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. I0220 18:44:39.065977 24842 ts_manager.cc:267] Unset tserver state for 493be4abe83d46c1829e3d5cfc68a9de from MAINTENANCE_MODE W0220 18:44:39.191025 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. W0220 18:44:39.191184 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. W0220 18:44:39.247881 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. I0220 18:44:39.266496 24772 heartbeater.cc:455] Master 127.23.135.190:45127 requested a full tablet report, sending... W0220 18:44:39.275825 24309 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. W0220 18:44:39.283480 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. W0220 18:44:39.404278 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. W0220 18:44:39.702219 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. W0220 18:44:39.764932 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. W0220 18:44:39.769989 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. I0220 18:44:39.793959 24420 heartbeater.cc:455] Master 127.23.135.190:45127 requested a full tablet report, sending... W0220 18:44:39.813084 24309 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. I0220 18:44:39.814759 24375 consensus_queue.cc:229] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1299, Committed index: 1299, Last appended: 4.1303, Last appended by leader: 16, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1304 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } } I0220 18:44:39.814813 24373 consensus_queue.cc:229] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1300, Committed index: 1300, Last appended: 1.1302, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1303 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } } I0220 18:44:39.816385 24500 raft_consensus.cc:1210] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 4 FOLLOWER]: Refusing update from remote peer c8a15b6a57804d05ada89b15de898a87: Log matching property violated. Preceding OpId in replica: term: 4 index: 1301. Preceding OpId from leader: term: 4 index: 1304. (index mismatch) I0220 18:44:39.816933 24499 raft_consensus.cc:1210] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Refusing update from remote peer c8a15b6a57804d05ada89b15de898a87: Log matching property violated. Preceding OpId in replica: term: 1 index: 1300. Preceding OpId from leader: term: 1 index: 1303. (index mismatch) I0220 18:44:39.817811 25086 consensus_queue.cc:1005] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1304, Last known committed idx: 1299, Time since last communication: 0.000s I0220 18:44:39.818032 25086 consensus_queue.cc:1005] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1303, Last known committed idx: 1300, Time since last communication: 0.000s W0220 18:44:39.824370 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0220 18:44:39.824414 24309 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0220 18:44:39.825038 25086 raft_consensus.cc:2878] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 [term 4 LEADER]: Committing config change with OpId 4.1304: config changed from index -1 to 1304, NON_VOTER 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132) added. New config: { opid_index: 1304 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } } } I0220 18:44:39.826252 25055 raft_consensus.cc:2878] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 [term 1 LEADER]: Committing config change with OpId 1.1303: config changed from index -1 to 1303, NON_VOTER 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132) added. New config: { opid_index: 1303 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } } } I0220 18:44:39.827468 24544 heartbeater.cc:455] Master 127.23.135.190:45127 requested a full tablet report, sending... I0220 18:44:39.830711 24500 raft_consensus.cc:2878] T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa [term 4 FOLLOWER]: Committing config change with OpId 4.1304: config changed from index -1 to 1304, NON_VOTER 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132) added. New config: { opid_index: 1304 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } } } I0220 18:44:39.832433 24500 raft_consensus.cc:2878] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa [term 1 FOLLOWER]: Committing config change with OpId 1.1303: config changed from index -1 to 1303, NON_VOTER 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132) added. New config: { opid_index: 1303 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } } } W0220 18:44:39.832844 24308 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132:45837): Couldn't send request to peer 609fb788e999458bb50fe5ede9fbe320. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ca7a33c93d548f49d2a4b1eeca470ca. This is attempt 1: this message will repeat every 5th retry. W0220 18:44:39.832950 24308 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132:45837): Couldn't send request to peer 609fb788e999458bb50fe5ede9fbe320. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5b2eb9c654f49df828ad8a096c62921. This is attempt 1: this message will repeat every 5th retry. I0220 18:44:39.835232 24828 catalog_manager.cc:3709] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2ca7a33c93d548f49d2a4b1eeca470ca with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0220 18:44:39.835839 24846 catalog_manager.cc:4173] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 reported cstate change: config changed from index -1 to 1304, NON_VOTER 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132) added. New cstate: current_term: 4 leader_uuid: "c8a15b6a57804d05ada89b15de898a87" committed_config { opid_index: 1304 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0220 18:44:39.845439 24501 consensus_queue.cc:229] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1302, Committed index: 1302, Last appended: 1.1303, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1304 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } } I0220 18:44:39.845485 24499 consensus_queue.cc:229] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1304, Committed index: 1304, Last appended: 2.1304, Last appended by leader: 16, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1305 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } } I0220 18:44:39.846864 24375 raft_consensus.cc:1210] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87 [term 2 FOLLOWER]: Refusing update from remote peer bccadedc7a11414384f1d4875bf49aaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 1304. Preceding OpId from leader: term: 2 index: 1305. (index mismatch) I0220 18:44:39.847196 24373 raft_consensus.cc:1210] T a7a069dabd5b4df28954b6833f23ab31 P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Refusing update from remote peer bccadedc7a11414384f1d4875bf49aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1302. Preceding OpId from leader: term: 1 index: 1304. (index mismatch) I0220 18:44:39.847213 25083 consensus_queue.cc:1005] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1305, Last known committed idx: 1304, Time since last communication: 0.000s I0220 18:44:39.847307 24498 consensus_queue.cc:229] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1303, Committed index: 1303, Last appended: 1.1303, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1304 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } } I0220 18:44:39.847733 25083 consensus_queue.cc:1005] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1304, Last known committed idx: 1300, Time since last communication: 0.000s W0220 18:44:39.847730 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0220 18:44:39.847769 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0220 18:44:39.848260 24497 consensus_queue.cc:229] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1303, Committed index: 1303, Last appended: 2.1303, Last appended by leader: 16, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1304 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } } I0220 18:44:39.848476 24375 raft_consensus.cc:1210] T c0ca382ac98b46948b2b18c2d691a364 P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Refusing update from remote peer bccadedc7a11414384f1d4875bf49aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1303. Preceding OpId from leader: term: 1 index: 1304. (index mismatch) W0220 18:44:39.848575 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0220 18:44:39.849407 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0220 18:44:39.850245 24375 raft_consensus.cc:1210] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [term 2 FOLLOWER]: Refusing update from remote peer bccadedc7a11414384f1d4875bf49aaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 1303. Preceding OpId from leader: term: 2 index: 1304. (index mismatch) I0220 18:44:39.850267 25083 consensus_queue.cc:1005] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1304, Last known committed idx: 1303, Time since last communication: 0.000s I0220 18:44:39.850630 25072 raft_consensus.cc:2878] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa [term 2 LEADER]: Committing config change with OpId 2.1305: config changed from index -1 to 1305, NON_VOTER 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132) added. New config: { opid_index: 1305 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } } } I0220 18:44:39.853411 25083 consensus_queue.cc:1005] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1304, Last known committed idx: 1303, Time since last communication: 0.000s I0220 18:44:39.853762 24375 raft_consensus.cc:2878] T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87 [term 2 FOLLOWER]: Committing config change with OpId 2.1305: config changed from index -1 to 1305, NON_VOTER 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132) added. New config: { opid_index: 1305 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } } } I0220 18:44:39.854040 25081 raft_consensus.cc:2878] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa [term 1 LEADER]: Committing config change with OpId 1.1304: config changed from index -1 to 1304, NON_VOTER 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132) added. New config: { opid_index: 1304 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } } } W0220 18:44:39.854511 24432 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132:45837): Couldn't send request to peer 609fb788e999458bb50fe5ede9fbe320. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 116ca55e12164875ad0a87183fe531bc. This is attempt 1: this message will repeat every 5th retry. W0220 18:44:39.854696 24432 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132:45837): Couldn't send request to peer 609fb788e999458bb50fe5ede9fbe320. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7a069dabd5b4df28954b6833f23ab31. This is attempt 1: this message will repeat every 5th retry. I0220 18:44:39.855265 24376 raft_consensus.cc:2878] T a7a069dabd5b4df28954b6833f23ab31 P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Committing config change with OpId 1.1304: config changed from index -1 to 1304, NON_VOTER 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132) added. New config: { opid_index: 1304 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } } } W0220 18:44:39.856313 24432 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132:45837): Couldn't send request to peer 609fb788e999458bb50fe5ede9fbe320. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0ca382ac98b46948b2b18c2d691a364. This is attempt 1: this message will repeat every 5th retry. W0220 18:44:39.856475 24432 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132:45837): Couldn't send request to peer 609fb788e999458bb50fe5ede9fbe320. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 827aa98f727b4da6a5015391b4da1960. This is attempt 1: this message will repeat every 5th retry. I0220 18:44:39.856946 25098 raft_consensus.cc:2878] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa [term 2 LEADER]: Committing config change with OpId 2.1304: config changed from index -1 to 1304, NON_VOTER 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132) added. New config: { opid_index: 1304 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } } } I0220 18:44:39.857241 24374 raft_consensus.cc:2878] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 [term 2 FOLLOWER]: Committing config change with OpId 2.1304: config changed from index -1 to 1304, NON_VOTER 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132) added. New config: { opid_index: 1304 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } } } I0220 18:44:39.857411 25084 raft_consensus.cc:2878] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa [term 1 LEADER]: Committing config change with OpId 1.1304: config changed from index -1 to 1304, NON_VOTER 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132) added. New config: { opid_index: 1304 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } } } I0220 18:44:39.858060 24373 raft_consensus.cc:2878] T c0ca382ac98b46948b2b18c2d691a364 P c8a15b6a57804d05ada89b15de898a87 [term 1 FOLLOWER]: Committing config change with OpId 1.1304: config changed from index -1 to 1304, NON_VOTER 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132) added. New config: { opid_index: 1304 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } } } I0220 18:44:39.905639 25116 ts_tablet_manager.cc:720] T 2ca7a33c93d548f49d2a4b1eeca470ca P 609fb788e999458bb50fe5ede9fbe320: Initiating tablet copy from peer c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621) I0220 18:44:39.905910 25116 tablet_copy_client.cc:244] T 2ca7a33c93d548f49d2a4b1eeca470ca P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Beginning tablet copy session from remote peer at address 127.23.135.130:40621 I0220 18:44:39.909715 24397 tablet_copy_service.cc:135] P c8a15b6a57804d05ada89b15de898a87: Received BeginTabletCopySession request for tablet 2ca7a33c93d548f49d2a4b1eeca470ca from peer 609fb788e999458bb50fe5ede9fbe320 ({username='slave'} at 127.23.135.132:58407) I0220 18:44:39.909796 24397 tablet_copy_service.cc:156] P c8a15b6a57804d05ada89b15de898a87: Beginning new tablet copy session on tablet 2ca7a33c93d548f49d2a4b1eeca470ca from peer 609fb788e999458bb50fe5ede9fbe320 at {username='slave'} at 127.23.135.132:58407: session id = 609fb788e999458bb50fe5ede9fbe320-2ca7a33c93d548f49d2a4b1eeca470ca I0220 18:44:39.910815 24397 tablet_copy_source_session.cc:204] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87: Tablet Copy: opened 0 blocks and 1 log segments I0220 18:44:39.913719 25116 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 2ca7a33c93d548f49d2a4b1eeca470ca. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:39.918098 25116 tablet_copy_client.cc:580] T 2ca7a33c93d548f49d2a4b1eeca470ca P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Starting download of 0 data blocks... I0220 18:44:39.918223 25116 tablet_copy_client.cc:543] T 2ca7a33c93d548f49d2a4b1eeca470ca P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Starting download of 1 WAL segments... I0220 18:44:39.918680 25118 ts_tablet_manager.cc:720] T c5b2eb9c654f49df828ad8a096c62921 P 609fb788e999458bb50fe5ede9fbe320: Initiating tablet copy from peer c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621) I0220 18:44:39.918785 25118 tablet_copy_client.cc:244] T c5b2eb9c654f49df828ad8a096c62921 P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Beginning tablet copy session from remote peer at address 127.23.135.130:40621 I0220 18:44:39.919251 24396 tablet_copy_service.cc:135] P c8a15b6a57804d05ada89b15de898a87: Received BeginTabletCopySession request for tablet c5b2eb9c654f49df828ad8a096c62921 from peer 609fb788e999458bb50fe5ede9fbe320 ({username='slave'} at 127.23.135.132:58407) I0220 18:44:39.919324 24396 tablet_copy_service.cc:156] P c8a15b6a57804d05ada89b15de898a87: Beginning new tablet copy session on tablet c5b2eb9c654f49df828ad8a096c62921 from peer 609fb788e999458bb50fe5ede9fbe320 at {username='slave'} at 127.23.135.132:58407: session id = 609fb788e999458bb50fe5ede9fbe320-c5b2eb9c654f49df828ad8a096c62921 I0220 18:44:39.922818 25116 tablet_copy_client.cc:421] T 2ca7a33c93d548f49d2a4b1eeca470ca P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0220 18:44:39.929735 24842 catalog_manager.cc:4173] T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa reported cstate change: config changed from index -1 to 1303, NON_VOTER 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132) added. New cstate: current_term: 1 leader_uuid: "c8a15b6a57804d05ada89b15de898a87" committed_config { opid_index: 1303 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } } } I0220 18:44:39.929935 24842 catalog_manager.cc:4173] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa reported cstate change: config changed from index -1 to 1305, NON_VOTER 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132) added. New cstate: current_term: 2 leader_uuid: "bccadedc7a11414384f1d4875bf49aaa" committed_config { opid_index: 1305 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0220 18:44:39.930634 24828 catalog_manager.cc:3709] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c5b2eb9c654f49df828ad8a096c62921 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0220 18:44:39.930742 24396 tablet_copy_source_session.cc:204] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87: Tablet Copy: opened 0 blocks and 1 log segments I0220 18:44:39.932072 25116 tablet_bootstrap.cc:441] T 2ca7a33c93d548f49d2a4b1eeca470ca P 609fb788e999458bb50fe5ede9fbe320: Bootstrap starting. I0220 18:44:39.935379 24828 catalog_manager.cc:3709] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 116ca55e12164875ad0a87183fe531bc with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0220 18:44:39.938042 24846 catalog_manager.cc:4173] T a7a069dabd5b4df28954b6833f23ab31 P c8a15b6a57804d05ada89b15de898a87 reported cstate change: config changed from index -1 to 1304, NON_VOTER 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132) added. New cstate: current_term: 1 leader_uuid: "bccadedc7a11414384f1d4875bf49aaa" committed_config { opid_index: 1304 OBSOLETE_local: false peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } } } I0220 18:44:39.938197 24846 catalog_manager.cc:4173] T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87 reported cstate change: config changed from index -1 to 1304, NON_VOTER 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132) added. New cstate: current_term: 2 leader_uuid: "bccadedc7a11414384f1d4875bf49aaa" committed_config { opid_index: 1304 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } } } I0220 18:44:39.941776 25118 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet c5b2eb9c654f49df828ad8a096c62921. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:39.944320 25118 tablet_copy_client.cc:580] T c5b2eb9c654f49df828ad8a096c62921 P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Starting download of 0 data blocks... I0220 18:44:39.944541 25118 tablet_copy_client.cc:543] T c5b2eb9c654f49df828ad8a096c62921 P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Starting download of 1 WAL segments... I0220 18:44:39.947279 25118 tablet_copy_client.cc:421] T c5b2eb9c654f49df828ad8a096c62921 P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0220 18:44:39.948977 25119 ts_tablet_manager.cc:720] T a7a069dabd5b4df28954b6833f23ab31 P 609fb788e999458bb50fe5ede9fbe320: Initiating tablet copy from peer bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:39.950703 25119 tablet_copy_client.cc:244] T a7a069dabd5b4df28954b6833f23ab31 P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Beginning tablet copy session from remote peer at address 127.23.135.131:46803 I0220 18:44:39.958348 24521 tablet_copy_service.cc:135] P bccadedc7a11414384f1d4875bf49aaa: Received BeginTabletCopySession request for tablet a7a069dabd5b4df28954b6833f23ab31 from peer 609fb788e999458bb50fe5ede9fbe320 ({username='slave'} at 127.23.135.132:60685) I0220 18:44:39.958433 24521 tablet_copy_service.cc:156] P bccadedc7a11414384f1d4875bf49aaa: Beginning new tablet copy session on tablet a7a069dabd5b4df28954b6833f23ab31 from peer 609fb788e999458bb50fe5ede9fbe320 at {username='slave'} at 127.23.135.132:60685: session id = 609fb788e999458bb50fe5ede9fbe320-a7a069dabd5b4df28954b6833f23ab31 I0220 18:44:39.959805 25120 ts_tablet_manager.cc:720] T c0ca382ac98b46948b2b18c2d691a364 P 609fb788e999458bb50fe5ede9fbe320: Initiating tablet copy from peer bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:39.959987 25120 tablet_copy_client.cc:244] T c0ca382ac98b46948b2b18c2d691a364 P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Beginning tablet copy session from remote peer at address 127.23.135.131:46803 I0220 18:44:39.961061 24520 tablet_copy_service.cc:135] P bccadedc7a11414384f1d4875bf49aaa: Received BeginTabletCopySession request for tablet c0ca382ac98b46948b2b18c2d691a364 from peer 609fb788e999458bb50fe5ede9fbe320 ({username='slave'} at 127.23.135.132:60685) I0220 18:44:39.961278 24520 tablet_copy_service.cc:156] P bccadedc7a11414384f1d4875bf49aaa: Beginning new tablet copy session on tablet c0ca382ac98b46948b2b18c2d691a364 from peer 609fb788e999458bb50fe5ede9fbe320 at {username='slave'} at 127.23.135.132:60685: session id = 609fb788e999458bb50fe5ede9fbe320-c0ca382ac98b46948b2b18c2d691a364 I0220 18:44:39.963773 25121 ts_tablet_manager.cc:720] T 827aa98f727b4da6a5015391b4da1960 P 609fb788e999458bb50fe5ede9fbe320: Initiating tablet copy from peer bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:39.963877 25121 tablet_copy_client.cc:244] T 827aa98f727b4da6a5015391b4da1960 P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Beginning tablet copy session from remote peer at address 127.23.135.131:46803 I0220 18:44:39.964288 24519 tablet_copy_service.cc:135] P bccadedc7a11414384f1d4875bf49aaa: Received BeginTabletCopySession request for tablet 827aa98f727b4da6a5015391b4da1960 from peer 609fb788e999458bb50fe5ede9fbe320 ({username='slave'} at 127.23.135.132:60685) I0220 18:44:39.964331 24519 tablet_copy_service.cc:156] P bccadedc7a11414384f1d4875bf49aaa: Beginning new tablet copy session on tablet 827aa98f727b4da6a5015391b4da1960 from peer 609fb788e999458bb50fe5ede9fbe320 at {username='slave'} at 127.23.135.132:60685: session id = 609fb788e999458bb50fe5ede9fbe320-827aa98f727b4da6a5015391b4da1960 I0220 18:44:39.967803 25122 ts_tablet_manager.cc:720] T 116ca55e12164875ad0a87183fe531bc P 609fb788e999458bb50fe5ede9fbe320: Initiating tablet copy from peer bccadedc7a11414384f1d4875bf49aaa (127.23.135.131:46803) I0220 18:44:39.967916 25122 tablet_copy_client.cc:244] T 116ca55e12164875ad0a87183fe531bc P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Beginning tablet copy session from remote peer at address 127.23.135.131:46803 I0220 18:44:39.968407 24518 tablet_copy_service.cc:135] P bccadedc7a11414384f1d4875bf49aaa: Received BeginTabletCopySession request for tablet 116ca55e12164875ad0a87183fe531bc from peer 609fb788e999458bb50fe5ede9fbe320 ({username='slave'} at 127.23.135.132:60685) I0220 18:44:39.968457 24518 tablet_copy_service.cc:156] P bccadedc7a11414384f1d4875bf49aaa: Beginning new tablet copy session on tablet 116ca55e12164875ad0a87183fe531bc from peer 609fb788e999458bb50fe5ede9fbe320 at {username='slave'} at 127.23.135.132:60685: session id = 609fb788e999458bb50fe5ede9fbe320-116ca55e12164875ad0a87183fe531bc I0220 18:44:39.969107 24518 tablet_copy_source_session.cc:204] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa: Tablet Copy: opened 0 blocks and 1 log segments I0220 18:44:40.019332 25116 log.cc:828] T 2ca7a33c93d548f49d2a4b1eeca470ca P 609fb788e999458bb50fe5ede9fbe320: Log is configured to *not* fsync() on all Append() calls I0220 18:44:40.021832 24521 tablet_copy_source_session.cc:204] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa: Tablet Copy: opened 0 blocks and 1 log segments I0220 18:44:40.022193 24519 tablet_copy_source_session.cc:204] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa: Tablet Copy: opened 0 blocks and 1 log segments I0220 18:44:40.022785 24520 tablet_copy_source_session.cc:204] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa: Tablet Copy: opened 0 blocks and 1 log segments I0220 18:44:40.024022 24828 catalog_manager.cc:3709] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a7a069dabd5b4df28954b6833f23ab31 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0220 18:44:40.024163 24828 catalog_manager.cc:3709] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 827aa98f727b4da6a5015391b4da1960 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0220 18:44:40.024248 24828 catalog_manager.cc:3709] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c0ca382ac98b46948b2b18c2d691a364 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0220 18:44:40.024631 25121 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 827aa98f727b4da6a5015391b4da1960. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:40.026578 25121 tablet_copy_client.cc:580] T 827aa98f727b4da6a5015391b4da1960 P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Starting download of 0 data blocks... I0220 18:44:40.026669 25121 tablet_copy_client.cc:543] T 827aa98f727b4da6a5015391b4da1960 P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Starting download of 1 WAL segments... I0220 18:44:40.027083 25120 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet c0ca382ac98b46948b2b18c2d691a364. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:40.027998 25119 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet a7a069dabd5b4df28954b6833f23ab31. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:40.028400 24842 catalog_manager.cc:4173] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa reported cstate change: config changed from index -1 to 1304, NON_VOTER 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132) added. New cstate: current_term: 1 leader_uuid: "bccadedc7a11414384f1d4875bf49aaa" committed_config { opid_index: 1304 OBSOLETE_local: false peers { permanent_uuid: "493be4abe83d46c1829e3d5cfc68a9de" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 35559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a15b6a57804d05ada89b15de898a87" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 40621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bccadedc7a11414384f1d4875bf49aaa" member_type: VOTER last_known_addr { host: "127.23.135.131" port: 46803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "609fb788e999458bb50fe5ede9fbe320" member_type: NON_VOTER last_known_addr { host: "127.23.135.132" port: 45837 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0220 18:44:40.029186 25120 tablet_copy_client.cc:580] T c0ca382ac98b46948b2b18c2d691a364 P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Starting download of 0 data blocks... I0220 18:44:40.029269 25120 tablet_copy_client.cc:543] T c0ca382ac98b46948b2b18c2d691a364 P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Starting download of 1 WAL segments... I0220 18:44:40.029687 25119 tablet_copy_client.cc:580] T a7a069dabd5b4df28954b6833f23ab31 P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Starting download of 0 data blocks... I0220 18:44:40.029783 25119 tablet_copy_client.cc:543] T a7a069dabd5b4df28954b6833f23ab31 P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Starting download of 1 WAL segments... I0220 18:44:40.030092 25122 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 116ca55e12164875ad0a87183fe531bc. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:40.035763 25118 tablet_bootstrap.cc:441] T c5b2eb9c654f49df828ad8a096c62921 P 609fb788e999458bb50fe5ede9fbe320: Bootstrap starting. I0220 18:44:40.039685 25122 tablet_copy_client.cc:580] T 116ca55e12164875ad0a87183fe531bc P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Starting download of 0 data blocks... I0220 18:44:40.040042 25122 tablet_copy_client.cc:543] T 116ca55e12164875ad0a87183fe531bc P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Starting download of 1 WAL segments... I0220 18:44:40.047578 25120 tablet_copy_client.cc:421] T c0ca382ac98b46948b2b18c2d691a364 P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0220 18:44:40.048687 25121 tablet_copy_client.cc:421] T 827aa98f727b4da6a5015391b4da1960 P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0220 18:44:40.050959 25119 tablet_copy_client.cc:421] T a7a069dabd5b4df28954b6833f23ab31 P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0220 18:44:40.051757 25122 tablet_copy_client.cc:421] T 116ca55e12164875ad0a87183fe531bc P 609fb788e999458bb50fe5ede9fbe320: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0220 18:44:40.118793 25120 tablet_bootstrap.cc:441] T c0ca382ac98b46948b2b18c2d691a364 P 609fb788e999458bb50fe5ede9fbe320: Bootstrap starting. W0220 18:44:40.212414 24622 log.cc:921] Time spent T c5b2eb9c654f49df828ad8a096c62921 P bccadedc7a11414384f1d4875bf49aaa: Append to log took a long time: real 0.069s user 0.000s sys 0.000s W0220 18:44:40.212435 24616 log.cc:921] Time spent T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa: Append to log took a long time: real 0.080s user 0.000s sys 0.000s W0220 18:44:40.212512 24640 log.cc:921] Time spent T 827aa98f727b4da6a5015391b4da1960 P c8a15b6a57804d05ada89b15de898a87: Append to log took a long time: real 0.081s user 0.000s sys 0.000s W0220 18:44:40.212524 24777 log.cc:921] Time spent T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87: Append to log took a long time: real 0.075s user 0.000s sys 0.000s W0220 18:44:40.236903 24601 log.cc:921] Time spent T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87: Append to log took a long time: real 0.099s user 0.000s sys 0.000s W0220 18:44:40.236935 24615 log.cc:921] Time spent T 116ca55e12164875ad0a87183fe531bc P c8a15b6a57804d05ada89b15de898a87: Append to log took a long time: real 0.105s user 0.000s sys 0.000s W0220 18:44:40.237219 24630 log.cc:921] Time spent T a7a069dabd5b4df28954b6833f23ab31 P c8a15b6a57804d05ada89b15de898a87: Append to log took a long time: real 0.099s user 0.000s sys 0.000s W0220 18:44:40.237458 24635 log.cc:921] Time spent T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa: Append to log took a long time: real 0.098s user 0.000s sys 0.000s I0220 18:44:40.237684 25121 tablet_bootstrap.cc:441] T 827aa98f727b4da6a5015391b4da1960 P 609fb788e999458bb50fe5ede9fbe320: Bootstrap starting. W0220 18:44:40.238165 24613 log.cc:921] Time spent T c0ca382ac98b46948b2b18c2d691a364 P c8a15b6a57804d05ada89b15de898a87: Append to log took a long time: real 0.102s user 0.000s sys 0.000s W0220 18:44:40.238404 24597 log.cc:921] Time spent T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa: Append to log took a long time: real 0.106s user 0.000s sys 0.000s I0220 18:44:40.238904 25122 tablet_bootstrap.cc:441] T 116ca55e12164875ad0a87183fe531bc P 609fb788e999458bb50fe5ede9fbe320: Bootstrap starting. W0220 18:44:40.239081 24593 log.cc:921] Time spent T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa: Append to log took a long time: real 0.107s user 0.000s sys 0.000s W0220 18:44:40.239873 24783 log.cc:921] Time spent T 2ca7a33c93d548f49d2a4b1eeca470ca P bccadedc7a11414384f1d4875bf49aaa: Append to log took a long time: real 0.104s user 0.000s sys 0.000s I0220 18:44:40.240898 25119 tablet_bootstrap.cc:441] T a7a069dabd5b4df28954b6833f23ab31 P 609fb788e999458bb50fe5ede9fbe320: Bootstrap starting. W0220 18:44:40.245575 25116 log.cc:921] Time spent T 2ca7a33c93d548f49d2a4b1eeca470ca P 609fb788e999458bb50fe5ede9fbe320: Append to log took a long time: real 0.100s user 0.000s sys 0.000s W0220 18:44:40.312494 24433 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0220 18:44:40.331002 24309 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0220 18:44:40.339525 24433 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0220 18:44:40.341675 24309 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0220 18:44:40.375445 24433 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0220 18:44:40.400868 24433 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559): Couldn't send request to peer 493be4abe83d46c1829e3d5cfc68a9de. Status: Network error: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. I0220 18:44:40.404258 24094 meta_cache.cc:1038] Marking tablet server 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559) as failed. W0220 18:44:40.404322 24094 meta_cache.cc:234] Tablet c0ca382ac98b46948b2b18c2d691a364: Replica 493be4abe83d46c1829e3d5cfc68a9de (127.23.135.129:35559) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.23.135.129:35559: connect: Connection refused (error 111) W0220 18:44:40.433774 24308 consensus_peers.cc:472] T 2ca7a33c93d548f49d2a4b1eeca470ca P c8a15b6a57804d05ada89b15de898a87 -> Peer 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132:45837): Couldn't send request to peer 609fb788e999458bb50fe5ede9fbe320. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry. W0220 18:44:40.473489 24432 consensus_peers.cc:472] T c0ca382ac98b46948b2b18c2d691a364 P bccadedc7a11414384f1d4875bf49aaa -> Peer 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132:45837): Couldn't send request to peer 609fb788e999458bb50fe5ede9fbe320. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry. W0220 18:44:40.475618 24432 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132:45837): Couldn't send request to peer 609fb788e999458bb50fe5ede9fbe320. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry. W0220 18:44:40.479555 24308 consensus_peers.cc:472] T c5b2eb9c654f49df828ad8a096c62921 P c8a15b6a57804d05ada89b15de898a87 -> Peer 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132:45837): Couldn't send request to peer 609fb788e999458bb50fe5ede9fbe320. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry. W0220 18:44:40.511065 24432 consensus_peers.cc:472] T 116ca55e12164875ad0a87183fe531bc P bccadedc7a11414384f1d4875bf49aaa -> Peer 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132:45837): Couldn't send request to peer 609fb788e999458bb50fe5ede9fbe320. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry. W0220 18:44:40.521796 24432 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer 609fb788e999458bb50fe5ede9fbe320 (127.23.135.132:45837): Couldn't send request to peer 609fb788e999458bb50fe5ede9fbe320. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry. I0220 18:44:40.650558 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 24297 W0220 18:44:40.662233 24432 connection.cc:658] client connection to 127.23.135.130:40621 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) W0220 18:44:40.662318 24432 consensus_peers.cc:472] T 827aa98f727b4da6a5015391b4da1960 P bccadedc7a11414384f1d4875bf49aaa -> Peer c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621): Couldn't send request to peer c8a15b6a57804d05ada89b15de898a87. Status: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry. W0220 18:44:40.662348 24432 consensus_peers.cc:472] T a7a069dabd5b4df28954b6833f23ab31 P bccadedc7a11414384f1d4875bf49aaa -> Peer c8a15b6a57804d05ada89b15de898a87 (127.23.135.130:40621): Couldn't send request to peer c8a15b6a57804d05ada89b15de898a87. Status: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry. I0220 18:44:40.663369 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 24423 I0220 18:44:40.671377 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 24611 I0220 18:44:40.675232 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-master with pid 24819 2020-02-20T18:44:40Z chronyd exiting [ OK ] MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate (16434 ms) [----------] 1 test from MaintenanceModeRF3ITest (16434 ms total) [----------] 1 test from RollingRestartArgs/RollingRestartITest [ RUN ] RollingRestartArgs/RollingRestartITest.TestWorkloads/4 2020-02-20T18:44:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG) 2020-02-20T18:44:40Z Disabled control of system clock I0220 18:44:40.708346 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-master /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.23.135.190:41437 --webserver_interface=127.23.135.190 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --rpc_reuseport=true --location_mapping_cmd=/tmp/dist-test-taskDlYUX2/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/location-assignment.state --map /L0:4 --master_client_location_assignment_enabled=false with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:44:40.766209 25142 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:44:40.766346 25142 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:44:40.767406 25142 flags.cc:403] Enabled experimental flag: --ipki_ca_key_size=1024 W0220 18:44:40.767431 25142 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:44:40.767437 25142 flags.cc:403] Enabled experimental flag: --tsk_num_rsa_bits=512 W0220 18:44:40.767446 25142 flags.cc:403] Enabled experimental flag: --rpc_reuseport=true I0220 18:44:40.769052 25142 master_runner.cc:90] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/master-0/wal --location_mapping_cmd=/tmp/dist-test-taskDlYUX2/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/location-assignment.state --map /L0:4 --ipki_ca_key_size=1024 --master_client_location_assignment_enabled=false --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.23.135.190:41437 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/master-0/data/info.pb --webserver_interface=127.23.135.190 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.25142 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-master --enable_minidumps=false --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:44:40.769565 25142 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:44:40.770278 25142 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:44:40.775163 25142 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:44:40.776265 25142 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:44:40.777403 25142 hybrid_clock.cc:552] HybridClock initialized: now 1582224280777377 us; error 26 us; skew 500 ppm I0220 18:44:40.777987 25142 fs_manager.cc:259] Metadata directory not provided I0220 18:44:40.778021 25142 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:44:40.778224 25142 server_base.cc:502] Could not load existing FS layout: Not found: could not find a healthy instance file I0220 18:44:40.778246 25142 server_base.cc:503] Attempting to create new FS layout instead I0220 18:44:40.779798 25142 fs_manager.cc:613] Generated new instance metadata in path /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/master-0/data/instance: uuid: "8e190ebeef534f089bbf57460c6e4115" format_stamp: "Formatted at 2020-02-20 18:44:40 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:40.780068 25142 fs_manager.cc:613] Generated new instance metadata in path /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/master-0/wal/instance: uuid: "8e190ebeef534f089bbf57460c6e4115" format_stamp: "Formatted at 2020-02-20 18:44:40 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:40.781639 25142 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/master-0/data/data/block_manager_instance: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/master-0/data/data/block_manager_instance: No such file or directory (error 2) I0220 18:44:40.786219 25142 fs_manager.cc:514] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s I0220 18:44:40.789912 25142 fs_manager.cc:430] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0220 18:44:40.789997 25142 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/master-0/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/master-0/wal uuid: "8e190ebeef534f089bbf57460c6e4115" format_stamp: "Formatted at 2020-02-20 18:44:40 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:40.790091 25142 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/master-0/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:44:40.827190 25142 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:44:40.832237 25142 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.190:41437 I0220 18:44:40.833006 25142 webserver.cc:382] Webserver started at http://127.23.135.190:46301/ using document root and password file I0220 18:44:40.833509 25142 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/master-0/data/info.pb I0220 18:44:40.836001 25203 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:40.840240 25203 tablet_bootstrap.cc:441] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115: Bootstrap starting. I0220 18:44:40.841890 25203 tablet_bootstrap.cc:597] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115: No blocks or log segments found. Creating new log. I0220 18:44:40.842382 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-master as pid 25142 I0220 18:44:40.842602 25203 log.cc:828] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115: Log is configured to *not* fsync() on all Append() calls I0220 18:44:40.842811 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.23.135.129:0 --local_ip_for_outbound_sockets=127.23.135.129 --webserver_interface=127.23.135.129 --webserver_port=0 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} I0220 18:44:40.843902 25203 tablet_bootstrap.cc:441] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115: No bootstrap required, opened a new log I0220 18:44:40.847103 25203 raft_consensus.cc:346] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e190ebeef534f089bbf57460c6e4115" member_type: VOTER } I0220 18:44:40.847184 25203 raft_consensus.cc:372] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:40.847236 25203 raft_consensus.cc:726] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e190ebeef534f089bbf57460c6e4115, State: Initialized, Role: FOLLOWER I0220 18:44:40.847810 25203 consensus_queue.cc:252] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e190ebeef534f089bbf57460c6e4115" member_type: VOTER } I0220 18:44:40.847893 25203 raft_consensus.cc:386] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0220 18:44:40.847937 25203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0220 18:44:40.848013 25203 raft_consensus.cc:2983] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115 [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:40.848841 25203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e190ebeef534f089bbf57460c6e4115" member_type: VOTER } I0220 18:44:40.849331 25203 leader_election.cc:307] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e190ebeef534f089bbf57460c6e4115; no voters: I0220 18:44:40.849715 25203 leader_election.cc:293] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115 [CANDIDATE]: Term 1 election: Requested vote from peers I0220 18:44:40.850040 25203 sys_catalog.cc:365] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e190ebeef534f089bbf57460c6e4115" member_type: VOTER } } I0220 18:44:40.850081 25203 sys_catalog.cc:368] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115 [sys.catalog]: This master's current role is: FOLLOWER I0220 18:44:40.850766 25203 sys_catalog.cc:466] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115 [sys.catalog]: configured and running, proceeding with master startup. I0220 18:44:40.851107 25206 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115 [term 1 FOLLOWER]: Leader election won for term 1 I0220 18:44:40.851402 25206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115 [term 1 LEADER]: Becoming Leader. State: Replica: 8e190ebeef534f089bbf57460c6e4115, State: Running, Role: LEADER I0220 18:44:40.851766 25206 consensus_queue.cc:229] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e190ebeef534f089bbf57460c6e4115" member_type: VOTER } I0220 18:44:40.853276 25209 sys_catalog.cc:365] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e190ebeef534f089bbf57460c6e4115. Latest consensus state: current_term: 1 leader_uuid: "8e190ebeef534f089bbf57460c6e4115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e190ebeef534f089bbf57460c6e4115" member_type: VOTER } } I0220 18:44:40.853371 25209 sys_catalog.cc:368] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115 [sys.catalog]: This master's current role is: LEADER I0220 18:44:40.853276 25207 sys_catalog.cc:365] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e190ebeef534f089bbf57460c6e4115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e190ebeef534f089bbf57460c6e4115" member_type: VOTER } } I0220 18:44:40.853612 25207 sys_catalog.cc:368] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115 [sys.catalog]: This master's current role is: LEADER W0220 18:44:40.855749 25208 catalog_manager.cc:1176] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0220 18:44:40.855787 25208 catalog_manager.cc:648] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0220 18:44:40.855850 25211 catalog_manager.cc:1107] Loading table and tablet metadata into memory... I0220 18:44:40.856670 25211 catalog_manager.cc:1118] Initializing Kudu internal certificate authority... I0220 18:44:40.877651 25211 catalog_manager.cc:1002] Generated new certificate authority record I0220 18:44:40.878577 25211 catalog_manager.cc:1127] Loading token signing keys... I0220 18:44:40.888132 25211 catalog_manager.cc:4509] T 00000000000000000000000000000000 P 8e190ebeef534f089bbf57460c6e4115: Generated new TSK 0 I0220 18:44:40.888545 25211 catalog_manager.cc:1137] Initializing in-progress tserver states... WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:44:40.904314 25205 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:44:40.904505 25205 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:44:40.905772 25205 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:44:40.905864 25205 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.129 I0220 18:44:40.907701 25205 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.129:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/info.pb --webserver_interface=127.23.135.129 --webserver_port=0 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.25205 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.129 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:44:40.908274 25205 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:44:40.909066 25205 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:44:40.916065 25205 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:44:40.917040 25205 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:44:40.924387 25205 hybrid_clock.cc:552] HybridClock initialized: now 1582224280924344 us; error 49 us; skew 500 ppm I0220 18:44:40.925105 25205 fs_manager.cc:259] Metadata directory not provided I0220 18:44:40.925151 25205 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:44:40.925366 25205 server_base.cc:502] Could not load existing FS layout: Not found: could not find a healthy instance file I0220 18:44:40.925392 25205 server_base.cc:503] Attempting to create new FS layout instead I0220 18:44:40.927232 25205 fs_manager.cc:613] Generated new instance metadata in path /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/instance: uuid: "853707498adf425fb8395cb60f41c9dc" format_stamp: "Formatted at 2020-02-20 18:44:40 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:40.927554 25205 fs_manager.cc:613] Generated new instance metadata in path /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal/instance: uuid: "853707498adf425fb8395cb60f41c9dc" format_stamp: "Formatted at 2020-02-20 18:44:40 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:40.929718 25205 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/data/block_manager_instance: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/data/block_manager_instance: No such file or directory (error 2) I0220 18:44:40.936480 25205 fs_manager.cc:514] Time spent creating directory manager: real 0.009s user 0.006s sys 0.001s I0220 18:44:40.940929 25205 fs_manager.cc:430] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0220 18:44:40.941025 25205 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal uuid: "853707498adf425fb8395cb60f41c9dc" format_stamp: "Formatted at 2020-02-20 18:44:40 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:40.941114 25205 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:44:40.969607 25205 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:44:40.970669 25205 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0220 18:44:40.970705 25205 ts_tablet_manager.cc:415] Registered 0 tablets I0220 18:44:40.978523 25205 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.129:34551 I0220 18:44:40.979492 25205 webserver.cc:382] Webserver started at http://127.23.135.129:33607/ using document root and password file I0220 18:44:40.981050 25205 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/info.pb I0220 18:44:40.987933 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 25205 I0220 18:44:40.988430 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.23.135.130:0 --local_ip_for_outbound_sockets=127.23.135.130 --webserver_interface=127.23.135.130 --webserver_port=0 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} I0220 18:44:40.992614 25335 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:44:40.992769 25335 heartbeater.cc:406] Registering TS with master... I0220 18:44:40.993191 25335 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:44:41.065650 25163 ts_manager.cc:192] Registered new tserver with Master: 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:44:41.067028 25163 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.129:41291 WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:44:41.081988 25339 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:44:41.082123 25339 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:44:41.083029 25339 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:44:41.083088 25339 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.130 I0220 18:44:41.084519 25339 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.130:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/info.pb --webserver_interface=127.23.135.130 --webserver_port=0 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.25339 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.130 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:44:41.084959 25339 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:44:41.085551 25339 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:44:41.090184 25339 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:44:41.090956 25339 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:44:41.092114 25339 hybrid_clock.cc:552] HybridClock initialized: now 1582224281092088 us; error 26 us; skew 500 ppm I0220 18:44:41.092692 25339 fs_manager.cc:259] Metadata directory not provided I0220 18:44:41.092726 25339 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:44:41.092895 25339 server_base.cc:502] Could not load existing FS layout: Not found: could not find a healthy instance file I0220 18:44:41.092912 25339 server_base.cc:503] Attempting to create new FS layout instead I0220 18:44:41.094393 25339 fs_manager.cc:613] Generated new instance metadata in path /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/instance: uuid: "a6e352f77ef54932a47d9977e15d05f1" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:41.094651 25339 fs_manager.cc:613] Generated new instance metadata in path /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal/instance: uuid: "a6e352f77ef54932a47d9977e15d05f1" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:41.096125 25339 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/data/block_manager_instance: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/data/block_manager_instance: No such file or directory (error 2) I0220 18:44:41.100231 25339 fs_manager.cc:514] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s I0220 18:44:41.103394 25339 fs_manager.cc:430] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0220 18:44:41.103466 25339 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal uuid: "a6e352f77ef54932a47d9977e15d05f1" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:41.103545 25339 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:44:41.122681 25339 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:44:41.123476 25339 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0220 18:44:41.123507 25339 ts_tablet_manager.cc:415] Registered 0 tablets I0220 18:44:41.129142 25339 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.130:36293 I0220 18:44:41.129822 25339 webserver.cc:382] Webserver started at http://127.23.135.130:39793/ using document root and password file I0220 18:44:41.130242 25339 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/info.pb I0220 18:44:41.134789 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 25339 I0220 18:44:41.135226 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.23.135.131:0 --local_ip_for_outbound_sockets=127.23.135.131 --webserver_interface=127.23.135.131 --webserver_port=0 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} I0220 18:44:41.138634 25461 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:44:41.138761 25461 heartbeater.cc:406] Registering TS with master... I0220 18:44:41.139075 25461 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:44:41.193302 25163 ts_manager.cc:192] Registered new tserver with Master: a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293) I0220 18:44:41.194099 25163 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.130:51711 WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:44:41.205435 25464 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:44:41.205608 25464 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:44:41.206516 25464 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:44:41.206586 25464 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.131 I0220 18:44:41.208009 25464 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.131:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/info.pb --webserver_interface=127.23.135.131 --webserver_port=0 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.25464 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.131 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:44:41.208498 25464 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:44:41.209198 25464 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:44:41.214110 25464 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:44:41.214915 25464 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:44:41.216043 25464 hybrid_clock.cc:552] HybridClock initialized: now 1582224281216022 us; error 24 us; skew 500 ppm I0220 18:44:41.216574 25464 fs_manager.cc:259] Metadata directory not provided I0220 18:44:41.216612 25464 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:44:41.216763 25464 server_base.cc:502] Could not load existing FS layout: Not found: could not find a healthy instance file I0220 18:44:41.216778 25464 server_base.cc:503] Attempting to create new FS layout instead I0220 18:44:41.218256 25464 fs_manager.cc:613] Generated new instance metadata in path /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/instance: uuid: "e4c29ba8de7e4745a4e497d213d63742" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:41.218485 25464 fs_manager.cc:613] Generated new instance metadata in path /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal/instance: uuid: "e4c29ba8de7e4745a4e497d213d63742" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:41.219921 25464 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/data/block_manager_instance: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/data/block_manager_instance: No such file or directory (error 2) I0220 18:44:41.224274 25464 fs_manager.cc:514] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s I0220 18:44:41.227381 25464 fs_manager.cc:430] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0220 18:44:41.227453 25464 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal uuid: "e4c29ba8de7e4745a4e497d213d63742" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:41.227519 25464 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:44:41.265453 25464 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:44:41.266264 25464 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0220 18:44:41.266294 25464 ts_tablet_manager.cc:415] Registered 0 tablets I0220 18:44:41.273588 25464 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.131:45349 I0220 18:44:41.274343 25464 webserver.cc:382] Webserver started at http://127.23.135.131:35617/ using document root and password file I0220 18:44:41.274782 25464 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/info.pb I0220 18:44:41.280257 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 25464 I0220 18:44:41.280704 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data --block_manager=log --rpc_bind_addresses=127.23.135.132:0 --local_ip_for_outbound_sockets=127.23.135.132 --webserver_interface=127.23.135.132 --webserver_port=0 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} I0220 18:44:41.282863 25586 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:44:41.282989 25586 heartbeater.cc:406] Registering TS with master... I0220 18:44:41.283298 25586 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:44:41.339429 25163 ts_manager.cc:192] Registered new tserver with Master: e4c29ba8de7e4745a4e497d213d63742 (127.23.135.131:45349) I0220 18:44:41.340291 25163 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.131:32773 WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:44:41.356940 25589 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:44:41.357123 25589 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:44:41.358042 25589 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:44:41.358129 25589 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.132 I0220 18:44:41.359503 25589 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.132:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/info.pb --webserver_interface=127.23.135.132 --webserver_port=0 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.25589 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.132 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:44:41.360028 25589 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:44:41.360705 25589 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:44:41.365811 25589 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:44:41.366668 25589 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:44:41.367785 25589 hybrid_clock.cc:552] HybridClock initialized: now 1582224281367766 us; error 27 us; skew 500 ppm I0220 18:44:41.368293 25589 fs_manager.cc:259] Metadata directory not provided I0220 18:44:41.368324 25589 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:44:41.368496 25589 server_base.cc:502] Could not load existing FS layout: Not found: could not find a healthy instance file I0220 18:44:41.368513 25589 server_base.cc:503] Attempting to create new FS layout instead I0220 18:44:41.369858 25589 fs_manager.cc:613] Generated new instance metadata in path /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/instance: uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:41.370093 25589 fs_manager.cc:613] Generated new instance metadata in path /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal/instance: uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:41.371484 25589 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/data/block_manager_instance: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/data/block_manager_instance: No such file or directory (error 2) I0220 18:44:41.375895 25589 fs_manager.cc:514] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s I0220 18:44:41.379061 25589 fs_manager.cc:430] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0220 18:44:41.379146 25589 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:41.379230 25589 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:44:41.399873 25589 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:44:41.400715 25589 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0220 18:44:41.400744 25589 ts_tablet_manager.cc:415] Registered 0 tablets I0220 18:44:41.408241 25589 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.132:33963 I0220 18:44:41.408943 25589 webserver.cc:382] Webserver started at http://127.23.135.132:44317/ using document root and password file I0220 18:44:41.409363 25589 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/info.pb I0220 18:44:41.415678 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 25589 I0220 18:44:41.416790 25711 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:44:41.416913 25711 heartbeater.cc:406] Registering TS with master... I0220 18:44:41.417270 25711 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:44:41.445315 25163 ts_manager.cc:192] Registered new tserver with Master: 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) I0220 18:44:41.446202 24094 external_mini_cluster.cc:738] 4 TS(s) registered with all masters I0220 18:44:41.446278 25163 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.132:33083 I0220 18:44:41.453496 24094 test_util.cc:228] Using random seed: 1279280054 I0220 18:44:41.461108 25163 catalog_manager.cc:1472] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42758: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0220 18:44:41.476491 25398 tablet_service.cc:1067] Processing CreateTablet for tablet 7e19d7c23cc941e3a7751fe2478481ec (table=test-workload [id=84bdfb5537804e8c9b2cded47cde24ba]), partition=RANGE (key) PARTITION UNBOUNDED I0220 18:44:41.476601 25645 tablet_service.cc:1067] Processing CreateTablet for tablet 7e19d7c23cc941e3a7751fe2478481ec (table=test-workload [id=84bdfb5537804e8c9b2cded47cde24ba]), partition=RANGE (key) PARTITION UNBOUNDED I0220 18:44:41.477007 25645 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 7e19d7c23cc941e3a7751fe2478481ec. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:41.477030 25398 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 7e19d7c23cc941e3a7751fe2478481ec. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:41.477159 25264 tablet_service.cc:1067] Processing CreateTablet for tablet 7e19d7c23cc941e3a7751fe2478481ec (table=test-workload [id=84bdfb5537804e8c9b2cded47cde24ba]), partition=RANGE (key) PARTITION UNBOUNDED I0220 18:44:41.477587 25264 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 7e19d7c23cc941e3a7751fe2478481ec. 1 dirs total, 0 dirs full, 0 dirs failed I0220 18:44:41.481534 25735 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Bootstrap starting. I0220 18:44:41.482957 25735 tablet_bootstrap.cc:597] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: No blocks or log segments found. Creating new log. I0220 18:44:41.483608 25735 log.cc:828] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Log is configured to *not* fsync() on all Append() calls I0220 18:44:41.483927 25736 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Bootstrap starting. I0220 18:44:41.484314 25737 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Bootstrap starting. I0220 18:44:41.485553 25737 tablet_bootstrap.cc:597] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: No blocks or log segments found. Creating new log. I0220 18:44:41.486068 25737 log.cc:828] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Log is configured to *not* fsync() on all Append() calls I0220 18:44:41.487428 25736 tablet_bootstrap.cc:597] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: No blocks or log segments found. Creating new log. I0220 18:44:41.487947 25736 log.cc:828] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Log is configured to *not* fsync() on all Append() calls I0220 18:44:41.489321 25736 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: No bootstrap required, opened a new log I0220 18:44:41.489419 25737 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: No bootstrap required, opened a new log I0220 18:44:41.489475 25736 ts_tablet_manager.cc:1113] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s I0220 18:44:41.489559 25737 ts_tablet_manager.cc:1113] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s I0220 18:44:41.491384 25737 raft_consensus.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:41.491385 25736 raft_consensus.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:41.491436 25737 raft_consensus.cc:372] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:41.491439 25736 raft_consensus.cc:372] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:41.491470 25737 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853707498adf425fb8395cb60f41c9dc, State: Initialized, Role: FOLLOWER I0220 18:44:41.491477 25736 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e352f77ef54932a47d9977e15d05f1, State: Initialized, Role: FOLLOWER I0220 18:44:41.491852 25736 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:41.492128 25737 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:41.492234 25461 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:44:41.492377 25736 ts_tablet_manager.cc:1142] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s I0220 18:44:41.492595 25735 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: No bootstrap required, opened a new log I0220 18:44:41.492696 25335 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:44:41.492733 25735 ts_tablet_manager.cc:1113] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s I0220 18:44:41.492830 25737 ts_tablet_manager.cc:1142] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s I0220 18:44:41.495090 25735 raft_consensus.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:41.495159 25735 raft_consensus.cc:372] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0220 18:44:41.495190 25735 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c751174fe9e41a0a4d3ea1dac04a50e, State: Initialized, Role: FOLLOWER I0220 18:44:41.495559 25735 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:41.495987 25711 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:44:41.496119 25735 ts_tablet_manager.cc:1142] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s I0220 18:44:41.510257 25741 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:44:41.510392 25741 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:41.511195 25741 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963), 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:44:41.515602 25663 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "a6e352f77ef54932a47d9977e15d05f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" is_pre_election: true I0220 18:44:41.515874 25663 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e352f77ef54932a47d9977e15d05f1 in term 0. I0220 18:44:41.516163 25292 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "a6e352f77ef54932a47d9977e15d05f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "853707498adf425fb8395cb60f41c9dc" is_pre_election: true I0220 18:44:41.516314 25292 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e352f77ef54932a47d9977e15d05f1 in term 0. I0220 18:44:41.516358 25348 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c751174fe9e41a0a4d3ea1dac04a50e, a6e352f77ef54932a47d9977e15d05f1; no voters: I0220 18:44:41.516638 25741 raft_consensus.cc:2730] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0220 18:44:41.516672 25741 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0220 18:44:41.516691 25741 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:41.517448 25741 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:41.517673 25741 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 1 election: Requested vote from peers 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963), 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:44:41.518023 25292 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "a6e352f77ef54932a47d9977e15d05f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "853707498adf425fb8395cb60f41c9dc" I0220 18:44:41.518044 25663 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "a6e352f77ef54932a47d9977e15d05f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" I0220 18:44:41.518083 25292 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:41.518097 25663 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 0 FOLLOWER]: Advancing to term 1 I0220 18:44:41.518749 25292 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e352f77ef54932a47d9977e15d05f1 in term 1. I0220 18:44:41.518821 25663 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e352f77ef54932a47d9977e15d05f1 in term 1. I0220 18:44:41.519040 25348 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 853707498adf425fb8395cb60f41c9dc, a6e352f77ef54932a47d9977e15d05f1; no voters: I0220 18:44:41.519179 25741 raft_consensus.cc:2730] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 1 FOLLOWER]: Leader election won for term 1 I0220 18:44:41.519354 25741 raft_consensus.cc:686] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 1 LEADER]: Becoming Leader. State: Replica: a6e352f77ef54932a47d9977e15d05f1, State: Running, Role: LEADER I0220 18:44:41.519702 25741 consensus_queue.cc:229] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:41.522408 25161 catalog_manager.cc:4173] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 reported cstate change: term changed from 0 to 1, leader changed from to a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130). New cstate: current_term: 1 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } health_report { overall_health: UNKNOWN } } } I0220 18:44:41.532997 24094 maintenance_mode-itest.cc:744] Restarting batch of 4 tservers: 853707498adf425fb8395cb60f41c9dc,a6e352f77ef54932a47d9977e15d05f1,e4c29ba8de7e4745a4e497d213d63742,8c751174fe9e41a0a4d3ea1dac04a50e I0220 18:44:41.575350 25292 raft_consensus.cc:1210] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 1 FOLLOWER]: Refusing update from remote peer a6e352f77ef54932a47d9977e15d05f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0220 18:44:41.575418 25663 raft_consensus.cc:1210] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 1 FOLLOWER]: Refusing update from remote peer a6e352f77ef54932a47d9977e15d05f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0220 18:44:41.576190 25741 consensus_queue.cc:1005] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0220 18:44:41.576505 25741 consensus_queue.cc:1005] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0220 18:44:41.593678 25762 mvcc.cc:203] Tried to move safe_time back from 6480790657324900352 to 6480790657106395136. Current Snapshot: MvccSnapshot[committed={T|T < 6480790657324900352}] I0220 18:44:41.593812 25763 mvcc.cc:203] Tried to move safe_time back from 6480790657324900352 to 6480790657106395136. Current Snapshot: MvccSnapshot[committed={T|T < 6480790657324900352}] I0220 18:44:41.620501 25764 mvcc.cc:203] Tried to move safe_time back from 6480790657324900352 to 6480790657106395136. Current Snapshot: MvccSnapshot[committed={T|T < 6480790657324900352 or (T in {6480790657324900352})}] I0220 18:44:41.738440 25169 ts_manager.cc:278] Set tserver state for a6e352f77ef54932a47d9977e15d05f1 to MAINTENANCE_MODE I0220 18:44:41.821740 25169 ts_manager.cc:278] Set tserver state for 853707498adf425fb8395cb60f41c9dc to MAINTENANCE_MODE I0220 18:44:41.864697 25169 ts_manager.cc:278] Set tserver state for e4c29ba8de7e4745a4e497d213d63742 to MAINTENANCE_MODE I0220 18:44:41.975206 25169 ts_manager.cc:278] Set tserver state for 8c751174fe9e41a0a4d3ea1dac04a50e to MAINTENANCE_MODE I0220 18:44:41.981495 25264 tablet_service.cc:1022] Tablet server 853707498adf425fb8395cb60f41c9dc set to quiescing I0220 18:44:41.981606 25264 tablet_service.cc:1029] Tablet server has 0 leaders and 0 scanners I0220 18:44:42.058512 25523 tablet_service.cc:1022] Tablet server e4c29ba8de7e4745a4e497d213d63742 set to quiescing I0220 18:44:42.058655 25523 tablet_service.cc:1029] Tablet server has 0 leaders and 0 scanners I0220 18:44:42.335525 25398 tablet_service.cc:1022] Tablet server a6e352f77ef54932a47d9977e15d05f1 set to quiescing I0220 18:44:42.335775 25398 tablet_service.cc:1029] Tablet server has 1 leaders and 0 scanners I0220 18:44:42.342399 25586 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:44:42.389433 25779 raft_consensus.cc:929] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: : Instructing follower 8c751174fe9e41a0a4d3ea1dac04a50e to start an election I0220 18:44:42.389605 25779 raft_consensus.cc:1017] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 1 LEADER]: Signalling peer 8c751174fe9e41a0a4d3ea1dac04a50e to start an election I0220 18:44:42.392917 25658 tablet_service.cc:1457] Received Run Leader Election RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" from {username='slave'} at 127.23.135.130:34413 I0220 18:44:42.393128 25658 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 1 FOLLOWER]: Starting forced leader election (received explicit request) I0220 18:44:42.393220 25658 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 1 FOLLOWER]: Advancing to term 2 I0220 18:44:42.394445 25658 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:42.396340 25658 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 2 election: Requested vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:44:42.401496 25658 raft_consensus.cc:1175] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 2 FOLLOWER]: Rejecting Update request from peer a6e352f77ef54932a47d9977e15d05f1 for earlier term 1. Current term is 2. Ops: [] I0220 18:44:42.402014 25745 consensus_queue.cc:1016] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 }, Status: INVALID_TERM, Last received: 1.100, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s I0220 18:44:42.402745 25779 raft_consensus.cc:2978] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 1 LEADER]: Stepping down as leader of term 1 I0220 18:44:42.402776 25779 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a6e352f77ef54932a47d9977e15d05f1, State: Running, Role: LEADER I0220 18:44:42.402956 25779 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 1.102, Last appended by leader: 102, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:42.403334 25779 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 1 FOLLOWER]: Advancing to term 2 W0220 18:44:42.403719 25762 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Cannot assign timestamp to transaction. Tablet is not in leader mode. Last heard from a leader: 0.001s ago. W0220 18:44:42.404686 25762 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Cannot assign timestamp to transaction. Tablet is not in leader mode. Last heard from a leader: 0.002s ago. W0220 18:44:42.407200 25626 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.407281 25627 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.414351 25236 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:42.415046 25292 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 2 candidate_status { last_received { term: 1 index: 100 } } ignore_live_leader: true dest_uuid: "853707498adf425fb8395cb60f41c9dc" I0220 18:44:42.415133 25418 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 2 candidate_status { last_received { term: 1 index: 100 } } ignore_live_leader: true dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" I0220 18:44:42.415179 25292 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 1 FOLLOWER]: Advancing to term 2 I0220 18:44:42.415272 25418 raft_consensus.cc:2356] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c751174fe9e41a0a4d3ea1dac04a50e for term 2 because replica has last-logged OpId of term: 1 index: 102, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 100. I0220 18:44:42.416260 25292 raft_consensus.cc:2356] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c751174fe9e41a0a4d3ea1dac04a50e for term 2 because replica has last-logged OpId of term: 1 index: 102, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 100. W0220 18:44:42.416487 25236 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:42.416626 25598 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c751174fe9e41a0a4d3ea1dac04a50e; no voters: 853707498adf425fb8395cb60f41c9dc, a6e352f77ef54932a47d9977e15d05f1 I0220 18:44:42.423862 25859 raft_consensus.cc:2675] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority W0220 18:44:42.425179 25366 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.425295 25369 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.432961 25627 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.433974 25627 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.443483 25236 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.444563 25236 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.453399 25369 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.453445 25366 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.463606 25627 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.464671 25626 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.473441 25236 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.476004 25235 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.485491 25369 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.489708 25369 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.497107 25627 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.503770 25628 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.514979 25235 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.518465 25235 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.530716 25369 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.534901 25369 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:42.540098 25645 tablet_service.cc:1022] Tablet server 8c751174fe9e41a0a4d3ea1dac04a50e set to quiescing I0220 18:44:42.540202 25645 tablet_service.cc:1029] Tablet server has 0 leaders and 0 scanners W0220 18:44:42.549461 25627 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.549552 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.565408 25236 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.565485 25235 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.584425 25369 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.586205 25369 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.605880 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.607971 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.628876 25235 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.628929 25236 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.652108 25369 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.652621 25779 raft_consensus.cc:659] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: failed to trigger leader election: Illegal state: leader elections are disabled W0220 18:44:42.653928 25369 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.676138 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.677029 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.700959 25235 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.701932 25235 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.706095 25870 raft_consensus.cc:659] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: failed to trigger leader election: Illegal state: leader elections are disabled W0220 18:44:42.725980 25369 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.726522 25366 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.750247 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.754209 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.776000 25235 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.782559 25235 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.806727 25366 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.814281 25366 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.825469 25859 raft_consensus.cc:659] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: failed to trigger leader election: Illegal state: leader elections are disabled W0220 18:44:42.834085 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.844460 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.864523 25235 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.871994 25235 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.898715 25366 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.902788 25366 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.928443 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.933452 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.962307 25235 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.965420 25235 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:42.996449 25366 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.001407 25366 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.034157 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.038167 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.071156 25235 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.075448 25235 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.110299 25366 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.113355 25366 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.147143 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.149039 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.185001 25235 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.187288 25235 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.224206 25366 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.225971 25366 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.267833 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.267925 25627 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.310778 25235 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.311795 25235 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37802: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.355971 25369 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.356070 25366 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.400076 25627 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.400760 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:43.436054 25398 tablet_service.cc:1022] Tablet server a6e352f77ef54932a47d9977e15d05f1 set to quiescing I0220 18:44:43.436152 25398 tablet_service.cc:1029] Tablet server has 0 leaders and 0 scanners I0220 18:44:43.441759 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 25205 W0220 18:44:43.445912 25721 connection.cc:658] client connection to 127.23.135.129:34551 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) W0220 18:44:43.445968 25721 meta_cache.cc:234] Tablet 7e19d7c23cc941e3a7751fe2478481ec: Replica 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) has failed: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) I0220 18:44:43.446347 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.23.135.129:34551 --local_ip_for_outbound_sockets=127.23.135.129 --webserver_port=33607 --webserver_interface=127.23.135.129 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:44:43.490368 25882 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:44:43.490532 25882 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:44:43.491080 25366 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.491598 25882 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:44:43.491691 25882 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.129 I0220 18:44:43.493000 25882 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.129:34551 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/info.pb --webserver_interface=127.23.135.129 --webserver_port=33607 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.25882 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.129 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:44:43.493458 25882 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:44:43.494069 25882 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:44:43.498896 25882 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:44:43.499658 25882 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:44:43.500793 25882 hybrid_clock.cc:552] HybridClock initialized: now 1582224283500774 us; error 20 us; skew 500 ppm I0220 18:44:43.501300 25882 fs_manager.cc:259] Metadata directory not provided I0220 18:44:43.501351 25882 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:44:43.505905 25882 fs_manager.cc:410] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s I0220 18:44:43.508888 25882 fs_manager.cc:430] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0220 18:44:43.509289 25882 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal uuid: "853707498adf425fb8395cb60f41c9dc" format_stamp: "Formatted at 2020-02-20 18:44:40 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:43.509380 25882 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0220 18:44:43.537685 25366 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34514: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.540114 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:43.540405 25882 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:44:43.541283 25882 ts_tablet_manager.cc:383] Loading tablet metadata (0/1 complete) I0220 18:44:43.543465 25882 ts_tablet_manager.cc:395] Loaded tablet metadata (1 total tablets, 1 live tablets) I0220 18:44:43.543496 25882 ts_tablet_manager.cc:401] Registering tablets (0/1 complete) I0220 18:44:43.544508 25882 ts_tablet_manager.cc:415] Registered 1 tablets I0220 18:44:43.544819 25900 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Bootstrap starting. I0220 18:44:43.553762 25882 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.129:34551 I0220 18:44:43.554684 25882 webserver.cc:382] Webserver started at http://127.23.135.129:33607/ using document root and password file I0220 18:44:43.555228 25882 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/info.pb I0220 18:44:43.560628 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 25882 I0220 18:44:43.560717 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 25339 I0220 18:44:43.563531 26005 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:44:43.563685 26005 heartbeater.cc:406] Registering TS with master... I0220 18:44:43.564015 26005 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:44:43.565219 25163 ts_manager.cc:192] Re-registered known tserver with Master: 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:44:43.566028 25163 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.129:59863 I0220 18:44:43.568325 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.23.135.130:36293 --local_ip_for_outbound_sockets=127.23.135.130 --webserver_port=39793 --webserver_interface=127.23.135.130 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} I0220 18:44:43.570911 25900 log.cc:828] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Log is configured to *not* fsync() on all Append() calls W0220 18:44:43.572871 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.604900 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.623576 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:44:43.627694 26009 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:44:43.627861 26009 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:44:43.629029 26009 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:44:43.629114 26009 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.130 I0220 18:44:43.630949 26009 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.130:36293 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/info.pb --webserver_interface=127.23.135.130 --webserver_port=39793 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.26009 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.130 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:44:43.631466 26009 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:44:43.632272 26009 file_cache.cc:476] Constructed file cache file cache with capacity 419430 W0220 18:44:43.635058 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:43.638882 26009 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:44:43.639847 26009 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:44:43.640991 26009 hybrid_clock.cc:552] HybridClock initialized: now 1582224283640961 us; error 25 us; skew 500 ppm I0220 18:44:43.641614 26009 fs_manager.cc:259] Metadata directory not provided I0220 18:44:43.641666 26009 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:44:43.648214 26009 fs_manager.cc:410] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s I0220 18:44:43.652284 26009 fs_manager.cc:430] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0220 18:44:43.652807 26009 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal uuid: "a6e352f77ef54932a47d9977e15d05f1" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:43.652933 26009 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0220 18:44:43.661556 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.691399 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:43.704622 26009 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:44:43.705449 26009 ts_tablet_manager.cc:383] Loading tablet metadata (0/1 complete) I0220 18:44:43.708076 26009 ts_tablet_manager.cc:395] Loaded tablet metadata (1 total tablets, 1 live tablets) I0220 18:44:43.708104 26009 ts_tablet_manager.cc:401] Registering tablets (0/1 complete) I0220 18:44:43.709118 26009 ts_tablet_manager.cc:415] Registered 1 tablets W0220 18:44:43.711849 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:43.711987 26028 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Bootstrap starting. I0220 18:44:43.720029 26009 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.130:36293 I0220 18:44:43.720898 26009 webserver.cc:382] Webserver started at http://127.23.135.130:39793/ using document root and password file I0220 18:44:43.721451 26009 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/info.pb I0220 18:44:43.723428 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 26009 I0220 18:44:43.723505 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 25464 I0220 18:44:43.729487 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.23.135.131:45349 --local_ip_for_outbound_sockets=127.23.135.131 --webserver_port=35617 --webserver_interface=127.23.135.131 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} I0220 18:44:43.735572 26133 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:44:43.735774 26133 heartbeater.cc:406] Registering TS with master... I0220 18:44:43.736202 26133 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:44:43.737721 25163 ts_manager.cc:192] Re-registered known tserver with Master: a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293) I0220 18:44:43.738772 25163 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.130:59561 I0220 18:44:43.742357 26028 log.cc:828] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Log is configured to *not* fsync() on all Append() calls W0220 18:44:43.773409 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:44:43.785362 26136 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true I0220 18:44:43.785490 25900 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=102 overwritten=0 applied=101 ignored=0} inserts{seen=5000 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates W0220 18:44:43.785516 26136 flags.cc:403] Enabled unsafe flag: --never_fsync=true I0220 18:44:43.785900 25900 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Bootstrap complete. W0220 18:44:43.786511 26136 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:44:43.786584 26136 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.131 I0220 18:44:43.787225 25900 ts_tablet_manager.cc:1113] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Time spent bootstrapping tablet: real 0.243s user 0.234s sys 0.000s I0220 18:44:43.788297 26136 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.131:45349 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/info.pb --webserver_interface=127.23.135.131 --webserver_port=35617 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.26136 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.131 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:44:43.788765 26136 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:44:43.789433 26136 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:44:43.790292 25900 raft_consensus.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 2 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } W0220 18:44:43.790720 25625 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39588: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:43.791481 25900 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853707498adf425fb8395cb60f41c9dc, State: Initialized, Role: FOLLOWER I0220 18:44:43.791980 25900 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 101, Last appended: 1.102, Last appended by leader: 102, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:43.792358 26005 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:44:43.793009 25900 ts_tablet_manager.cc:1142] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s I0220 18:44:43.796169 26136 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:44:43.796948 26136 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:44:43.798068 26136 hybrid_clock.cc:552] HybridClock initialized: now 1582224283798043 us; error 23 us; skew 500 ppm I0220 18:44:43.798561 26136 fs_manager.cc:259] Metadata directory not provided I0220 18:44:43.798604 26136 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:44:43.803352 26136 fs_manager.cc:410] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s I0220 18:44:43.806269 26136 fs_manager.cc:430] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0220 18:44:43.806691 26136 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal uuid: "e4c29ba8de7e4745a4e497d213d63742" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:43.806787 26136 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:44:43.829514 26136 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:44:43.830325 26136 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0220 18:44:43.830355 26136 ts_tablet_manager.cc:415] Registered 0 tablets I0220 18:44:43.835616 26136 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.131:45349 I0220 18:44:43.837175 26136 webserver.cc:382] Webserver started at http://127.23.135.131:35617/ using document root and password file I0220 18:44:43.837651 26136 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/info.pb I0220 18:44:43.845788 26261 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:44:43.845917 26261 heartbeater.cc:406] Registering TS with master... I0220 18:44:43.846220 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 26136 I0220 18:44:43.846292 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 25589 I0220 18:44:43.846313 26261 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:44:43.847573 25163 ts_manager.cc:192] Re-registered known tserver with Master: e4c29ba8de7e4745a4e497d213d63742 (127.23.135.131:45349) I0220 18:44:43.848507 25163 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.131:39705 I0220 18:44:43.851375 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data --block_manager=log --rpc_bind_addresses=127.23.135.132:33963 --local_ip_for_outbound_sockets=127.23.135.132 --webserver_port=44317 --webserver_interface=127.23.135.132 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} I0220 18:44:43.874908 25750 meta_cache.cc:1038] Marking tablet server 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) as failed. WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:44:43.923166 26264 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:44:43.923341 26264 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:44:43.924522 26264 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:44:43.924605 26264 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.132 I0220 18:44:43.926576 26264 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.132:33963 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/info.pb --webserver_interface=127.23.135.132 --webserver_port=44317 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.26264 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.132 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:44:43.927213 26264 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:44:43.928125 26264 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:44:43.934826 26264 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:44:43.935956 26264 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:44:43.937098 26264 hybrid_clock.cc:552] HybridClock initialized: now 1582224283937057 us; error 38 us; skew 500 ppm I0220 18:44:43.937754 26264 fs_manager.cc:259] Metadata directory not provided I0220 18:44:43.937813 26264 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:44:43.944469 26264 fs_manager.cc:410] Time spent opening directory manager: real 0.006s user 0.002s sys 0.004s I0220 18:44:43.948777 26264 fs_manager.cc:430] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0220 18:44:43.949323 26264 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:43.949440 26264 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0220 18:44:43.951140 25921 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:43.955230 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:43.992328 26264 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:44:43.993405 26264 ts_tablet_manager.cc:383] Loading tablet metadata (0/1 complete) I0220 18:44:43.996335 26264 ts_tablet_manager.cc:395] Loaded tablet metadata (1 total tablets, 1 live tablets) I0220 18:44:43.996374 26264 ts_tablet_manager.cc:401] Registering tablets (0/1 complete) I0220 18:44:43.997702 26264 ts_tablet_manager.cc:415] Registered 1 tablets I0220 18:44:44.004479 26282 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Bootstrap starting. I0220 18:44:44.006265 26264 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.132:33963 I0220 18:44:44.007112 26264 webserver.cc:382] Webserver started at http://127.23.135.132:44317/ using document root and password file I0220 18:44:44.008615 26264 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/info.pb I0220 18:44:44.015208 26028 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=102 overwritten=0 applied=101 ignored=0} inserts{seen=5000 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0220 18:44:44.015580 26028 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Bootstrap complete. I0220 18:44:44.017171 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 26264 I0220 18:44:44.020735 26028 ts_tablet_manager.cc:1113] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Time spent bootstrapping tablet: real 0.309s user 0.224s sys 0.008s I0220 18:44:44.020877 26387 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:44:44.021046 26387 heartbeater.cc:406] Registering TS with master... I0220 18:44:44.021435 26387 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:44:44.022737 25163 ts_manager.cc:192] Re-registered known tserver with Master: 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) I0220 18:44:44.023561 26028 raft_consensus.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:44.023921 25163 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.132:49661 I0220 18:44:44.027076 26282 log.cc:828] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Log is configured to *not* fsync() on all Append() calls I0220 18:44:44.036741 26028 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e352f77ef54932a47d9977e15d05f1, State: Initialized, Role: FOLLOWER I0220 18:44:44.037160 26028 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 101, Last appended: 1.102, Last appended by leader: 102, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:44.037534 26133 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:44:44.037720 26028 ts_tablet_manager.cc:1142] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s I0220 18:44:44.047607 26143 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:44:44.047845 26143 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:44.048820 26143 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) I0220 18:44:44.054456 26090 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 3 candidate_status { last_received { term: 1 index: 102 } } ignore_live_leader: false dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" is_pre_election: true I0220 18:44:44.054651 26090 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 853707498adf425fb8395cb60f41c9dc in term 2. I0220 18:44:44.055115 25893 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 853707498adf425fb8395cb60f41c9dc, a6e352f77ef54932a47d9977e15d05f1; no voters: I0220 18:44:44.055270 26338 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 3 candidate_status { last_received { term: 1 index: 102 } } ignore_live_leader: false dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" is_pre_election: true I0220 18:44:44.055382 26143 raft_consensus.cc:2730] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 2 FOLLOWER]: Leader pre-election won for term 3 I0220 18:44:44.055408 26143 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0220 18:44:44.055426 26143 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 2 FOLLOWER]: Advancing to term 3 W0220 18:44:44.056088 25891 leader_election.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963): Illegal state: must be running to vote when last-logged opid is not known I0220 18:44:44.056881 26143 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:44.057159 26143 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 3 election: Requested vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) I0220 18:44:44.057543 26090 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 3 candidate_status { last_received { term: 1 index: 102 } } ignore_live_leader: false dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" I0220 18:44:44.057610 26090 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 2 FOLLOWER]: Advancing to term 3 I0220 18:44:44.057615 26338 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 3 candidate_status { last_received { term: 1 index: 102 } } ignore_live_leader: false dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" W0220 18:44:44.057925 25891 leader_election.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 3 election: Tablet error from VoteRequest() call to peer 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963): Illegal state: must be running to vote when last-logged opid is not known I0220 18:44:44.059003 26090 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 853707498adf425fb8395cb60f41c9dc in term 3. I0220 18:44:44.059296 25893 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 853707498adf425fb8395cb60f41c9dc, a6e352f77ef54932a47d9977e15d05f1; no voters: 8c751174fe9e41a0a4d3ea1dac04a50e I0220 18:44:44.059459 26143 raft_consensus.cc:2730] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 3 FOLLOWER]: Leader election won for term 3 I0220 18:44:44.059692 26143 raft_consensus.cc:686] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 3 LEADER]: Becoming Leader. State: Replica: 853707498adf425fb8395cb60f41c9dc, State: Running, Role: LEADER I0220 18:44:44.060055 26143 consensus_queue.cc:229] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 1.102, Last appended by leader: 102, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:44.062762 25163 catalog_manager.cc:4173] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc reported cstate change: term changed from 1 to 3, leader changed from a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130) to 853707498adf425fb8395cb60f41c9dc (127.23.135.129). New cstate: current_term: 3 leader_uuid: "853707498adf425fb8395cb60f41c9dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } health_report { overall_health: HEALTHY } } } W0220 18:44:44.087199 25891 consensus_peers.cc:472] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc -> Peer 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963): Couldn't send request to peer 8c751174fe9e41a0a4d3ea1dac04a50e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry. I0220 18:44:44.087206 26090 raft_consensus.cc:1210] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 3 FOLLOWER]: Refusing update from remote peer 853707498adf425fb8395cb60f41c9dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 3 index: 104. (index mismatch) I0220 18:44:44.088063 26143 consensus_queue.cc:1005] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 101, Time since last communication: 0.000s I0220 18:44:44.368479 26282 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Bootstrap replayed 1/1 log segments. Stats: ops{read=100 overwritten=0 applied=100 ignored=0} inserts{seen=4950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0220 18:44:44.368877 26282 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Bootstrap complete. I0220 18:44:44.369834 26282 ts_tablet_manager.cc:1113] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Time spent bootstrapping tablet: real 0.366s user 0.208s sys 0.013s I0220 18:44:44.376580 26282 raft_consensus.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:44.376662 26282 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c751174fe9e41a0a4d3ea1dac04a50e, State: Initialized, Role: FOLLOWER I0220 18:44:44.377135 26282 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 100, Last appended: 1.100, Last appended by leader: 100, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:44.377581 26387 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:44:44.378335 26282 ts_tablet_manager.cc:1142] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s I0220 18:44:44.385903 26338 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 2 FOLLOWER]: Advancing to term 3 I0220 18:44:44.387591 26338 raft_consensus.cc:1210] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 3 FOLLOWER]: Refusing update from remote peer 853707498adf425fb8395cb60f41c9dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 100. Preceding OpId from leader: term: 1 index: 102. (index mismatch) I0220 18:44:44.388425 26396 consensus_queue.cc:1007] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 100, Time since last communication: 0.000s I0220 18:44:44.466096 26441 mvcc.cc:203] Tried to move safe_time back from 6480790668852928512 to 6480790667511885824. Current Snapshot: MvccSnapshot[committed={T|T < 6480790668812996608}] I0220 18:44:44.850239 26261 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:44:50.224159 25169 ts_manager.cc:267] Unset tserver state for 853707498adf425fb8395cb60f41c9dc from MAINTENANCE_MODE I0220 18:44:50.226778 25163 ts_manager.cc:267] Unset tserver state for e4c29ba8de7e4745a4e497d213d63742 from MAINTENANCE_MODE I0220 18:44:50.241910 25170 ts_manager.cc:267] Unset tserver state for 8c751174fe9e41a0a4d3ea1dac04a50e from MAINTENANCE_MODE I0220 18:44:50.244928 25164 ts_manager.cc:267] Unset tserver state for a6e352f77ef54932a47d9977e15d05f1 from MAINTENANCE_MODE I0220 18:44:50.424641 26387 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:44:50.426807 26005 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:44:50.511867 25164 ts_manager.cc:278] Set tserver state for 853707498adf425fb8395cb60f41c9dc to MAINTENANCE_MODE I0220 18:44:50.678628 25164 ts_manager.cc:278] Set tserver state for 8c751174fe9e41a0a4d3ea1dac04a50e to MAINTENANCE_MODE I0220 18:44:50.698791 25164 ts_manager.cc:278] Set tserver state for e4c29ba8de7e4745a4e497d213d63742 to MAINTENANCE_MODE I0220 18:44:50.765393 25937 tablet_service.cc:1022] Tablet server 853707498adf425fb8395cb60f41c9dc set to quiescing I0220 18:44:50.765503 25937 tablet_service.cc:1029] Tablet server has 1 leaders and 0 scanners I0220 18:44:50.766070 26408 raft_consensus.cc:929] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: : Instructing follower a6e352f77ef54932a47d9977e15d05f1 to start an election I0220 18:44:50.766222 26408 raft_consensus.cc:1017] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 3 LEADER]: Signalling peer a6e352f77ef54932a47d9977e15d05f1 to start an election I0220 18:44:50.767000 26089 tablet_service.cc:1457] Received Run Leader Election RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" from {username='slave'} at 127.23.135.129:35989 I0220 18:44:50.767072 26089 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 3 FOLLOWER]: Starting forced leader election (received explicit request) I0220 18:44:50.767107 26089 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 3 FOLLOWER]: Advancing to term 4 I0220 18:44:50.769039 26089 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:50.769838 26089 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 4 election: Requested vote from peers 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963), 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:44:50.773579 26405 raft_consensus.cc:929] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: : Instructing follower 8c751174fe9e41a0a4d3ea1dac04a50e to start an election I0220 18:44:50.773653 26405 raft_consensus.cc:1017] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 3 LEADER]: Signalling peer 8c751174fe9e41a0a4d3ea1dac04a50e to start an election I0220 18:44:50.777817 26089 raft_consensus.cc:1175] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 4 FOLLOWER]: Rejecting Update request from peer 853707498adf425fb8395cb60f41c9dc for earlier term 3. Current term is 4. Ops: [] I0220 18:44:50.778167 25164 ts_manager.cc:278] Set tserver state for a6e352f77ef54932a47d9977e15d05f1 to MAINTENANCE_MODE I0220 18:44:50.778224 26408 consensus_queue.cc:1016] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 }, Status: INVALID_TERM, Last received: 3.1336, Next index: 1337, Last known committed idx: 1336, Time since last communication: 0.000s I0220 18:44:50.779044 26396 raft_consensus.cc:929] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: : Instructing follower 8c751174fe9e41a0a4d3ea1dac04a50e to start an election I0220 18:44:50.779109 26396 raft_consensus.cc:1017] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 3 LEADER]: Signalling peer 8c751174fe9e41a0a4d3ea1dac04a50e to start an election I0220 18:44:50.779181 26396 raft_consensus.cc:2978] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 3 LEADER]: Stepping down as leader of term 3 I0220 18:44:50.779197 26396 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 853707498adf425fb8395cb60f41c9dc, State: Running, Role: LEADER I0220 18:44:50.779326 26396 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1337, Committed index: 1337, Last appended: 3.1337, Last appended by leader: 1337, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:50.779443 26396 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 3 FOLLOWER]: Advancing to term 4 I0220 18:44:50.780179 26332 tablet_service.cc:1457] Received Run Leader Election RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" from {username='slave'} at 127.23.135.129:44379 I0220 18:44:50.780247 26332 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 3 FOLLOWER]: Starting forced leader election (received explicit request) I0220 18:44:50.780292 26332 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 3 FOLLOWER]: Advancing to term 4 I0220 18:44:50.781229 26332 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:50.782032 26332 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 4 election: Requested vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:44:50.784291 26338 tablet_service.cc:1457] Received Run Leader Election RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" from {username='slave'} at 127.23.135.129:44379 I0220 18:44:50.784345 26338 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 4 FOLLOWER]: Starting forced leader election (received explicit request) I0220 18:44:50.784369 26338 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 4 FOLLOWER]: Advancing to term 5 I0220 18:44:50.785143 26338 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:50.785356 26338 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 5 election: Requested vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:44:50.785925 26089 raft_consensus.cc:1175] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 4 FOLLOWER]: Rejecting Update request from peer 853707498adf425fb8395cb60f41c9dc for earlier term 3. Current term is 4. Ops: [] W0220 18:44:50.789120 26142 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Cannot assign timestamp to transaction. Tablet is not in leader mode. Last heard from a leader: 0.010s ago. W0220 18:44:50.790169 26142 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Cannot assign timestamp to transaction. Tablet is not in leader mode. Last heard from a leader: 0.011s ago. W0220 18:44:50.790673 25917 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:50.792712 26046 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:50.793570 25962 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "a6e352f77ef54932a47d9977e15d05f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 1336 } } ignore_live_leader: true dest_uuid: "853707498adf425fb8395cb60f41c9dc" I0220 18:44:50.793692 25962 raft_consensus.cc:2356] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a6e352f77ef54932a47d9977e15d05f1 for term 4 because replica has last-logged OpId of term: 3 index: 1337, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 1336. I0220 18:44:50.794734 26338 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "a6e352f77ef54932a47d9977e15d05f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 1336 } } ignore_live_leader: true dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" I0220 18:44:50.794809 26338 raft_consensus.cc:2314] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a6e352f77ef54932a47d9977e15d05f1 for earlier term 4. Current term is 5. I0220 18:44:50.795156 26019 leader_election.cc:403] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 4 election: Vote denied by peer 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) with higher term. Message: Invalid argument: T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a6e352f77ef54932a47d9977e15d05f1 for earlier term 4. Current term is 5. I0220 18:44:50.795176 26019 leader_election.cc:406] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term W0220 18:44:50.795840 26049 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:50.796382 26609 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 4 FOLLOWER]: Advancing to term 5 I0220 18:44:50.797243 26609 raft_consensus.cc:2675] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) with higher term. Message: Invalid argument: T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a6e352f77ef54932a47d9977e15d05f1 for earlier term 4. Current term is 5. I0220 18:44:50.799949 25962 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 4 candidate_status { last_received { term: 3 index: 1337 } } ignore_live_leader: true dest_uuid: "853707498adf425fb8395cb60f41c9dc" I0220 18:44:50.800917 25962 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c751174fe9e41a0a4d3ea1dac04a50e in term 4. I0220 18:44:50.801132 25961 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 5 candidate_status { last_received { term: 3 index: 1337 } } ignore_live_leader: true dest_uuid: "853707498adf425fb8395cb60f41c9dc" I0220 18:44:50.801177 25961 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 4 FOLLOWER]: Advancing to term 5 I0220 18:44:50.801297 26273 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 853707498adf425fb8395cb60f41c9dc, 8c751174fe9e41a0a4d3ea1dac04a50e; no voters: I0220 18:44:50.801934 25961 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c751174fe9e41a0a4d3ea1dac04a50e in term 5. I0220 18:44:50.802204 26273 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 853707498adf425fb8395cb60f41c9dc, 8c751174fe9e41a0a4d3ea1dac04a50e; no voters: I0220 18:44:50.802734 26610 raft_consensus.cc:2690] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won W0220 18:44:50.804942 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:50.805308 26612 raft_consensus.cc:2730] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 5 FOLLOWER]: Leader election won for term 5 I0220 18:44:50.805442 26612 raft_consensus.cc:686] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 5 LEADER]: Becoming Leader. State: Replica: 8c751174fe9e41a0a4d3ea1dac04a50e, State: Running, Role: LEADER I0220 18:44:50.806032 26612 consensus_queue.cc:229] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1337, Committed index: 1337, Last appended: 3.1337, Last appended by leader: 1337, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:50.799443 26089 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 4 candidate_status { last_received { term: 3 index: 1337 } } ignore_live_leader: true dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" I0220 18:44:50.806352 26089 raft_consensus.cc:2314] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c751174fe9e41a0a4d3ea1dac04a50e for earlier term 4. Current term is 5. W0220 18:44:50.806412 26050 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:50.799515 26090 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 5 candidate_status { last_received { term: 3 index: 1337 } } ignore_live_leader: true dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" I0220 18:44:50.807399 26090 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c751174fe9e41a0a4d3ea1dac04a50e in term 5. I0220 18:44:50.807901 26275 leader_election.cc:403] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 4 election: Vote denied by peer a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293) with higher term. Message: Invalid argument: T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c751174fe9e41a0a4d3ea1dac04a50e for earlier term 4. Current term is 5. I0220 18:44:50.809998 25164 catalog_manager.cc:4173] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e reported cstate change: term changed from 3 to 5, leader changed from 853707498adf425fb8395cb60f41c9dc (127.23.135.129) to 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132). New cstate: current_term: 5 leader_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } health_report { overall_health: UNKNOWN } } } I0220 18:44:50.813944 25961 raft_consensus.cc:1210] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 5 FOLLOWER]: Refusing update from remote peer 8c751174fe9e41a0a4d3ea1dac04a50e: Log matching property violated. Preceding OpId in replica: term: 3 index: 1337. Preceding OpId from leader: term: 5 index: 1340. (index mismatch) I0220 18:44:50.814083 26090 raft_consensus.cc:1210] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 5 FOLLOWER]: Refusing update from remote peer 8c751174fe9e41a0a4d3ea1dac04a50e: Log matching property violated. Preceding OpId in replica: term: 3 index: 1336. Preceding OpId from leader: term: 5 index: 1340. (index mismatch) I0220 18:44:50.814471 26614 consensus_queue.cc:1005] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [LEADER]: Connected to new peer: Peer: permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1338, Last known committed idx: 1337, Time since last communication: 0.000s I0220 18:44:50.816790 26613 consensus_queue.cc:1005] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1338, Last known committed idx: 1336, Time since last communication: 0.000s I0220 18:44:50.826126 26142 mvcc.cc:203] Tried to move safe_time back from 6480790695167942656 to 6480790695145992192. Current Snapshot: MvccSnapshot[committed={T|T < 6480790695167942656 or (T in {6480790695167942656})}] I0220 18:44:50.839520 26133 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:44:50.847038 26393 mvcc.cc:203] Tried to move safe_time back from 6480790695182331904 to 6480790695145992192. Current Snapshot: MvccSnapshot[committed={T|T < 6480790695182331904}] I0220 18:44:50.860944 26261 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:44:51.008612 26306 tablet_service.cc:1022] Tablet server 8c751174fe9e41a0a4d3ea1dac04a50e set to quiescing I0220 18:44:51.008774 26306 tablet_service.cc:1029] Tablet server has 1 leaders and 0 scanners I0220 18:44:51.023384 26614 raft_consensus.cc:929] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: : Instructing follower a6e352f77ef54932a47d9977e15d05f1 to start an election I0220 18:44:51.023546 26614 raft_consensus.cc:1017] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 5 LEADER]: Signalling peer a6e352f77ef54932a47d9977e15d05f1 to start an election I0220 18:44:51.024660 26089 tablet_service.cc:1457] Received Run Leader Election RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" from {username='slave'} at 127.23.135.132:41059 I0220 18:44:51.024722 26089 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 5 FOLLOWER]: Starting forced leader election (received explicit request) I0220 18:44:51.024751 26089 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 5 FOLLOWER]: Advancing to term 6 I0220 18:44:51.025741 26089 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:51.026015 26089 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 6 election: Requested vote from peers 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963), 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:44:51.026676 26338 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "a6e352f77ef54932a47d9977e15d05f1" candidate_term: 6 candidate_status { last_received { term: 5 index: 1366 } } ignore_live_leader: true dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" I0220 18:44:51.026738 26338 raft_consensus.cc:2978] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 5 LEADER]: Stepping down as leader of term 5 I0220 18:44:51.026755 26338 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8c751174fe9e41a0a4d3ea1dac04a50e, State: Running, Role: LEADER I0220 18:44:51.026880 26338 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1366, Committed index: 1366, Last appended: 5.1367, Last appended by leader: 1367, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:51.026923 26338 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 5 FOLLOWER]: Advancing to term 6 I0220 18:44:51.027959 26338 raft_consensus.cc:2356] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a6e352f77ef54932a47d9977e15d05f1 for term 6 because replica has last-logged OpId of term: 5 index: 1367, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 1366. I0220 18:44:51.029780 26089 raft_consensus.cc:1175] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 6 FOLLOWER]: Rejecting Update request from peer 8c751174fe9e41a0a4d3ea1dac04a50e for earlier term 5. Current term is 6. Ops: [] I0220 18:44:51.030397 26198 tablet_service.cc:1022] Tablet server e4c29ba8de7e4745a4e497d213d63742 set to quiescing I0220 18:44:51.030515 26198 tablet_service.cc:1029] Tablet server has 0 leaders and 0 scanners W0220 18:44:51.030699 26441 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:51.030963 25961 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "a6e352f77ef54932a47d9977e15d05f1" candidate_term: 6 candidate_status { last_received { term: 5 index: 1366 } } ignore_live_leader: true dest_uuid: "853707498adf425fb8395cb60f41c9dc" I0220 18:44:51.031391 25961 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 5 FOLLOWER]: Advancing to term 6 W0220 18:44:51.032325 26441 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Cannot assign timestamp to transaction. Tablet is not in leader mode. Last heard from a leader: 0.007s ago. I0220 18:44:51.032457 25961 raft_consensus.cc:2356] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a6e352f77ef54932a47d9977e15d05f1 for term 6 because replica has last-logged OpId of term: 5 index: 1367, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 1366. I0220 18:44:51.033939 26019 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a6e352f77ef54932a47d9977e15d05f1; no voters: 853707498adf425fb8395cb60f41c9dc, 8c751174fe9e41a0a4d3ea1dac04a50e I0220 18:44:51.034215 26609 raft_consensus.cc:2675] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority I0220 18:44:51.034322 26070 tablet_service.cc:1022] Tablet server a6e352f77ef54932a47d9977e15d05f1 set to quiescing I0220 18:44:51.034430 26070 tablet_service.cc:1029] Tablet server has 0 leaders and 3 scanners W0220 18:44:51.036670 26050 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.038202 26046 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.042590 25917 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.044692 25917 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.050392 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.052408 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.057744 26047 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.058805 26046 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.065058 25917 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.065142 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.073844 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.078104 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.083779 26049 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.087683 26046 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.096340 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.097738 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.109359 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.111240 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.123590 26047 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.125762 26050 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.139194 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.139946 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.154609 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.157763 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.173081 26050 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.174208 26049 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.188973 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.191619 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.209491 26295 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.209512 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.230835 26046 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.230854 26050 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.251190 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.252593 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.273202 26295 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.275137 26295 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.294497 26046 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.298677 26049 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.319056 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.322705 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.328300 26645 raft_consensus.cc:659] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: failed to trigger leader election: Illegal state: leader elections are disabled W0220 18:44:51.344421 26295 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.346258 26295 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.368937 26047 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.373975 26047 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.376068 26615 raft_consensus.cc:659] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: failed to trigger leader election: Illegal state: leader elections are disabled W0220 18:44:51.397496 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.400470 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.426405 26295 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.427057 26295 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.453091 26609 raft_consensus.cc:659] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: failed to trigger leader election: Illegal state: leader elections are disabled W0220 18:44:51.453747 26047 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.453857 26046 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.482550 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.483453 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.516743 26295 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.516793 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.548645 26046 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.550611 26046 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.581400 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.582372 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.615459 26295 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.616052 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.652813 26047 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.652881 26046 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.686739 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.690752 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.722805 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.728883 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.761704 26046 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.768854 26046 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.801784 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.808835 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.841907 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.850298 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:51.875939 25937 tablet_service.cc:1022] Tablet server 853707498adf425fb8395cb60f41c9dc set to quiescing I0220 18:44:51.876046 25937 tablet_service.cc:1029] Tablet server has 0 leaders and 0 scanners W0220 18:44:51.884953 26046 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.892268 26046 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.929188 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.932102 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.975538 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:51.976346 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:52.017936 26046 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:52.018848 26046 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:52.061044 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:52.066221 25906 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37824: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:52.108681 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:52.112004 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:52.118991 26306 tablet_service.cc:1022] Tablet server 8c751174fe9e41a0a4d3ea1dac04a50e set to quiescing I0220 18:44:52.119081 26306 tablet_service.cc:1029] Tablet server has 0 leaders and 0 scanners I0220 18:44:52.136984 26070 tablet_service.cc:1022] Tablet server a6e352f77ef54932a47d9977e15d05f1 set to quiescing I0220 18:44:52.137087 26070 tablet_service.cc:1029] Tablet server has 0 leaders and 0 scanners I0220 18:44:52.142210 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 25882 I0220 18:44:52.147444 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.23.135.129:34551 --local_ip_for_outbound_sockets=127.23.135.129 --webserver_port=33607 --webserver_interface=127.23.135.129 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} W0220 18:44:52.157668 26046 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:52.159741 26046 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34556: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:44:52.191435 26679 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:44:52.191570 26679 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:44:52.192416 26679 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:44:52.192466 26679 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.129 I0220 18:44:52.193735 26679 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.129:34551 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/info.pb --webserver_interface=127.23.135.129 --webserver_port=33607 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.26679 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.129 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:44:52.194169 26679 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:44:52.194779 26679 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:44:52.199415 26679 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:44:52.200160 26679 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:44:52.201282 26679 hybrid_clock.cc:552] HybridClock initialized: now 1582224292201250 us; error 38 us; skew 500 ppm I0220 18:44:52.201779 26679 fs_manager.cc:259] Metadata directory not provided I0220 18:44:52.201812 26679 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory W0220 18:44:52.205102 25721 meta_cache.cc:234] Tablet 7e19d7c23cc941e3a7751fe2478481ec: Replica 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) has failed: Network error: Client connection negotiation failed: client connection to 127.23.135.129:34551: connect: Connection refused (error 111) I0220 18:44:52.206445 26679 fs_manager.cc:410] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s W0220 18:44:52.208797 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:52.209651 26679 fs_manager.cc:430] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0220 18:44:52.210057 26679 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal uuid: "853707498adf425fb8395cb60f41c9dc" format_stamp: "Formatted at 2020-02-20 18:44:40 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:52.210147 26679 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:44:52.239264 26679 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:44:52.240222 26679 ts_tablet_manager.cc:383] Loading tablet metadata (0/1 complete) I0220 18:44:52.242516 26679 ts_tablet_manager.cc:395] Loaded tablet metadata (1 total tablets, 1 live tablets) I0220 18:44:52.242548 26679 ts_tablet_manager.cc:401] Registering tablets (0/1 complete) I0220 18:44:52.243582 26679 ts_tablet_manager.cc:415] Registered 1 tablets I0220 18:44:52.243959 26697 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Bootstrap starting. I0220 18:44:52.250571 26679 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.129:34551 I0220 18:44:52.251399 26679 webserver.cc:382] Webserver started at http://127.23.135.129:33607/ using document root and password file I0220 18:44:52.252976 26679 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/info.pb I0220 18:44:52.261814 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 26679 I0220 18:44:52.261888 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 26009 I0220 18:44:52.262814 26802 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:44:52.262938 26802 heartbeater.cc:406] Registering TS with master... I0220 18:44:52.263317 26802 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:44:52.264645 25164 ts_manager.cc:192] Re-registered known tserver with Master: 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:44:52.265709 25164 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.129:41823 I0220 18:44:52.269471 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.23.135.130:36293 --local_ip_for_outbound_sockets=127.23.135.130 --webserver_port=39793 --webserver_interface=127.23.135.130 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} W0220 18:44:52.318856 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:44:52.360436 26807 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:44:52.360589 26807 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:44:52.361470 26807 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:44:52.361539 26807 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.130 I0220 18:44:52.362857 26807 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.130:36293 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/info.pb --webserver_interface=127.23.135.130 --webserver_port=39793 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.26807 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.130 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:44:52.363337 26807 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:44:52.368192 26807 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:44:52.377434 26807 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:44:52.378366 26807 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:44:52.379523 26807 hybrid_clock.cc:552] HybridClock initialized: now 1582224292379490 us; error 28 us; skew 500 ppm I0220 18:44:52.380111 26807 fs_manager.cc:259] Metadata directory not provided I0220 18:44:52.380146 26807 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:44:52.384819 26807 fs_manager.cc:410] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s I0220 18:44:52.387825 26807 fs_manager.cc:430] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0220 18:44:52.388352 26807 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal uuid: "a6e352f77ef54932a47d9977e15d05f1" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:52.388458 26807 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0220 18:44:52.417012 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:52.423065 26697 log.cc:828] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Log is configured to *not* fsync() on all Append() calls I0220 18:44:52.429766 26807 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:44:52.430574 26807 ts_tablet_manager.cc:383] Loading tablet metadata (0/1 complete) I0220 18:44:52.432955 26807 ts_tablet_manager.cc:395] Loaded tablet metadata (1 total tablets, 1 live tablets) I0220 18:44:52.432986 26807 ts_tablet_manager.cc:401] Registering tablets (0/1 complete) I0220 18:44:52.433976 26807 ts_tablet_manager.cc:415] Registered 1 tablets I0220 18:44:52.435932 26825 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Bootstrap starting. I0220 18:44:52.443517 26807 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.130:36293 I0220 18:44:52.444342 26807 webserver.cc:382] Webserver started at http://127.23.135.130:39793/ using document root and password file I0220 18:44:52.444850 26807 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/info.pb I0220 18:44:52.445724 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 26807 I0220 18:44:52.445804 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 26136 I0220 18:44:52.451599 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.23.135.131:45349 --local_ip_for_outbound_sockets=127.23.135.131 --webserver_port=35617 --webserver_interface=127.23.135.131 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} I0220 18:44:52.461189 26930 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:44:52.461320 26930 heartbeater.cc:406] Registering TS with master... I0220 18:44:52.461700 26930 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:44:52.462957 25164 ts_manager.cc:192] Re-registered known tserver with Master: a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293) I0220 18:44:52.464174 25164 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.130:55277 W0220 18:44:52.476107 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:44:52.545842 26933 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:44:52.545994 26933 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:44:52.546931 26933 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:44:52.547014 26933 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.131 I0220 18:44:52.552582 26933 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.131:45349 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/info.pb --webserver_interface=127.23.135.131 --webserver_port=35617 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.26933 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.131 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:44:52.553056 26933 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:44:52.553717 26933 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:44:52.562595 26933 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:44:52.563323 26933 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:44:52.578811 26933 hybrid_clock.cc:552] HybridClock initialized: now 1582224292578767 us; error 46 us; skew 500 ppm I0220 18:44:52.579555 26933 fs_manager.cc:259] Metadata directory not provided I0220 18:44:52.579680 26933 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:44:52.588469 26933 fs_manager.cc:410] Time spent opening directory manager: real 0.008s user 0.006s sys 0.000s W0220 18:44:52.591040 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:52.594885 26933 fs_manager.cc:430] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0220 18:44:52.595420 26933 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal uuid: "e4c29ba8de7e4745a4e497d213d63742" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:52.595532 26933 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:44:52.609973 26825 log.cc:828] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Log is configured to *not* fsync() on all Append() calls I0220 18:44:52.631057 26933 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:44:52.632087 26933 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0220 18:44:52.632123 26933 ts_tablet_manager.cc:415] Registered 0 tablets I0220 18:44:52.639526 26933 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.131:45349 I0220 18:44:52.640350 26933 webserver.cc:382] Webserver started at http://127.23.135.131:35617/ using document root and password file I0220 18:44:52.640920 26933 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/info.pb W0220 18:44:52.650015 26293 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39628: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:52.650748 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 26933 I0220 18:44:52.650823 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 26264 I0220 18:44:52.658428 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data --block_manager=log --rpc_bind_addresses=127.23.135.132:33963 --local_ip_for_outbound_sockets=127.23.135.132 --webserver_port=44317 --webserver_interface=127.23.135.132 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} I0220 18:44:52.667775 27056 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:44:52.667922 27056 heartbeater.cc:406] Registering TS with master... I0220 18:44:52.668323 27056 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:44:52.669227 25164 ts_manager.cc:192] Re-registered known tserver with Master: e4c29ba8de7e4745a4e497d213d63742 (127.23.135.131:45349) I0220 18:44:52.670089 25164 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.131:45113 I0220 18:44:52.674039 25750 meta_cache.cc:1038] Marking tablet server 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) as failed. I0220 18:44:52.677812 25752 meta_cache.cc:1038] Marking tablet server 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) as failed. WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:44:52.717383 27059 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:44:52.717732 27059 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:44:52.719048 27059 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:44:52.719204 27059 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.132 I0220 18:44:52.721344 27059 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.132:33963 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/info.pb --webserver_interface=127.23.135.132 --webserver_port=44317 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.27059 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.132 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:44:52.721905 27059 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:44:52.722698 27059 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:44:52.729280 27059 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:44:52.730305 27059 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:44:52.731458 27059 hybrid_clock.cc:552] HybridClock initialized: now 1582224292731433 us; error 27 us; skew 500 ppm I0220 18:44:52.732115 27059 fs_manager.cc:259] Metadata directory not provided I0220 18:44:52.732161 27059 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:44:52.738482 27059 fs_manager.cc:410] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s I0220 18:44:52.742556 27059 fs_manager.cc:430] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0220 18:44:52.743117 27059 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:44:52.743234 27059 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:44:52.773692 27059 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:44:52.774688 27059 ts_tablet_manager.cc:383] Loading tablet metadata (0/1 complete) I0220 18:44:52.777488 27059 ts_tablet_manager.cc:395] Loaded tablet metadata (1 total tablets, 1 live tablets) I0220 18:44:52.777523 27059 ts_tablet_manager.cc:401] Registering tablets (0/1 complete) I0220 18:44:52.778756 27059 ts_tablet_manager.cc:415] Registered 1 tablets I0220 18:44:52.779989 27076 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Bootstrap starting. I0220 18:44:52.789512 27059 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.132:33963 I0220 18:44:52.790467 27059 webserver.cc:382] Webserver started at http://127.23.135.132:44317/ using document root and password file I0220 18:44:52.791035 27059 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/info.pb I0220 18:44:52.793599 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 27059 I0220 18:44:52.810575 27181 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:44:52.810696 27181 heartbeater.cc:406] Registering TS with master... I0220 18:44:52.811025 27181 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:44:52.812290 25164 ts_manager.cc:192] Re-registered known tserver with Master: 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) I0220 18:44:52.813308 25164 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.132:48337 I0220 18:44:52.957751 27076 log.cc:828] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Log is configured to *not* fsync() on all Append() calls I0220 18:44:53.267454 26802 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:44:53.465950 26930 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:44:53.671823 27056 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:44:53.814997 27181 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:44:54.796242 26697 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=1367 overwritten=0 applied=1366 ignored=0} inserts{seen=68150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0220 18:44:54.796746 26697 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Bootstrap complete. I0220 18:44:54.798203 26697 ts_tablet_manager.cc:1113] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Time spent bootstrapping tablet: real 2.554s user 2.412s sys 0.048s I0220 18:44:54.801604 26697 raft_consensus.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 6 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:54.802842 26697 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853707498adf425fb8395cb60f41c9dc, State: Initialized, Role: FOLLOWER I0220 18:44:54.803287 26697 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1366, Last appended: 5.1367, Last appended by leader: 1367, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:54.804227 26697 ts_tablet_manager.cc:1142] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s W0220 18:44:54.983503 26719 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:54.986253 26719 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:55.056969 27219 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:44:55.057155 27219 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:55.058032 27219 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) I0220 18:44:55.064599 26883 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 7 candidate_status { last_received { term: 5 index: 1367 } } ignore_live_leader: false dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" is_pre_election: true I0220 18:44:55.065049 27119 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 7 candidate_status { last_received { term: 5 index: 1367 } } ignore_live_leader: false dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" is_pre_election: true W0220 18:44:55.065259 26690 leader_election.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 7 pre-election: Tablet error from VoteRequest() call to peer a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293): Illegal state: must be running to vote when last-logged opid is not known W0220 18:44:55.065588 26688 leader_election.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 7 pre-election: Tablet error from VoteRequest() call to peer 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963): Illegal state: must be running to vote when last-logged opid is not known I0220 18:44:55.065668 26688 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 853707498adf425fb8395cb60f41c9dc; no voters: 8c751174fe9e41a0a4d3ea1dac04a50e, a6e352f77ef54932a47d9977e15d05f1 I0220 18:44:55.065922 27219 raft_consensus.cc:2675] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority W0220 18:44:55.255429 26718 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:55.255502 26719 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:55.484572 27219 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:44:55.484699 27219 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:55.485004 27219 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) I0220 18:44:55.485687 27119 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 7 candidate_status { last_received { term: 5 index: 1367 } } ignore_live_leader: false dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" is_pre_election: true I0220 18:44:55.486016 26883 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 7 candidate_status { last_received { term: 5 index: 1367 } } ignore_live_leader: false dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" is_pre_election: true W0220 18:44:55.486073 26688 leader_election.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 7 pre-election: Tablet error from VoteRequest() call to peer 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963): Illegal state: must be running to vote when last-logged opid is not known W0220 18:44:55.486380 26690 leader_election.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 7 pre-election: Tablet error from VoteRequest() call to peer a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293): Illegal state: must be running to vote when last-logged opid is not known I0220 18:44:55.486429 26690 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 853707498adf425fb8395cb60f41c9dc; no voters: 8c751174fe9e41a0a4d3ea1dac04a50e, a6e352f77ef54932a47d9977e15d05f1 I0220 18:44:55.486660 27219 raft_consensus.cc:2675] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority W0220 18:44:55.533910 26718 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:55.537212 26718 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:55.822842 26708 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:55.824357 26708 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:55.875483 27219 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:44:55.875619 27219 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:55.876106 27219 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) I0220 18:44:55.876579 27119 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 7 candidate_status { last_received { term: 5 index: 1367 } } ignore_live_leader: false dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" is_pre_election: true I0220 18:44:55.876696 26883 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 7 candidate_status { last_received { term: 5 index: 1367 } } ignore_live_leader: false dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" is_pre_election: true W0220 18:44:55.876987 26688 leader_election.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 7 pre-election: Tablet error from VoteRequest() call to peer 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963): Illegal state: must be running to vote when last-logged opid is not known W0220 18:44:55.877152 26690 leader_election.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 7 pre-election: Tablet error from VoteRequest() call to peer a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293): Illegal state: must be running to vote when last-logged opid is not known I0220 18:44:55.877204 26690 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 853707498adf425fb8395cb60f41c9dc; no voters: 8c751174fe9e41a0a4d3ea1dac04a50e, a6e352f77ef54932a47d9977e15d05f1 I0220 18:44:55.877431 27219 raft_consensus.cc:2675] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority I0220 18:44:55.878480 26825 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=1366 overwritten=0 applied=1366 ignored=0} inserts{seen=68150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0220 18:44:55.878857 26825 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Bootstrap complete. I0220 18:44:55.880074 26825 ts_tablet_manager.cc:1113] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Time spent bootstrapping tablet: real 3.444s user 3.138s sys 0.095s I0220 18:44:55.882674 26825 raft_consensus.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:55.882746 26825 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e352f77ef54932a47d9977e15d05f1, State: Initialized, Role: FOLLOWER I0220 18:44:55.883240 26825 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1366, Last appended: 5.1366, Last appended by leader: 1366, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:55.883831 26825 ts_tablet_manager.cc:1142] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s W0220 18:44:56.019994 26833 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:56.021308 26833 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:56.118952 26709 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:56.125718 26711 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:56.229212 27225 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:44:56.229409 27225 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:56.230449 27225 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963), 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:44:56.236383 27119 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "a6e352f77ef54932a47d9977e15d05f1" candidate_term: 7 candidate_status { last_received { term: 5 index: 1366 } } ignore_live_leader: false dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" is_pre_election: true W0220 18:44:56.237000 26815 leader_election.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 7 pre-election: Tablet error from VoteRequest() call to peer 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963): Illegal state: must be running to vote when last-logged opid is not known I0220 18:44:56.240967 26753 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "a6e352f77ef54932a47d9977e15d05f1" candidate_term: 7 candidate_status { last_received { term: 5 index: 1366 } } ignore_live_leader: false dest_uuid: "853707498adf425fb8395cb60f41c9dc" is_pre_election: true I0220 18:44:56.241118 26753 raft_consensus.cc:2356] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a6e352f77ef54932a47d9977e15d05f1 for term 7 because replica has last-logged OpId of term: 5 index: 1367, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 1366. I0220 18:44:56.241554 26815 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a6e352f77ef54932a47d9977e15d05f1; no voters: 853707498adf425fb8395cb60f41c9dc, 8c751174fe9e41a0a4d3ea1dac04a50e I0220 18:44:56.241853 27225 raft_consensus.cc:2675] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority I0220 18:44:56.256876 27076 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1367 overwritten=0 applied=1366 ignored=0} inserts{seen=68150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0220 18:44:56.256924 27219 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:44:56.257045 27219 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:56.257361 27219 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) I0220 18:44:56.257552 27076 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Bootstrap complete. I0220 18:44:56.257761 27119 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 7 candidate_status { last_received { term: 5 index: 1367 } } ignore_live_leader: false dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" is_pre_election: true I0220 18:44:56.258047 26883 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 7 candidate_status { last_received { term: 5 index: 1367 } } ignore_live_leader: false dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" is_pre_election: true W0220 18:44:56.258147 26688 leader_election.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 7 pre-election: Tablet error from VoteRequest() call to peer 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963): Illegal state: must be running to vote when last-logged opid is not known I0220 18:44:56.258167 26883 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 853707498adf425fb8395cb60f41c9dc in term 6. I0220 18:44:56.258556 26690 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 853707498adf425fb8395cb60f41c9dc, a6e352f77ef54932a47d9977e15d05f1; no voters: 8c751174fe9e41a0a4d3ea1dac04a50e I0220 18:44:56.258713 27219 raft_consensus.cc:2730] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 6 FOLLOWER]: Leader pre-election won for term 7 I0220 18:44:56.258744 27219 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0220 18:44:56.258772 27219 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 6 FOLLOWER]: Advancing to term 7 I0220 18:44:56.258893 27076 ts_tablet_manager.cc:1113] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Time spent bootstrapping tablet: real 3.479s user 3.349s sys 0.072s I0220 18:44:56.260524 27219 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:56.260815 27219 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 7 election: Requested vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) I0220 18:44:56.261243 26883 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 7 candidate_status { last_received { term: 5 index: 1367 } } ignore_live_leader: false dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" I0220 18:44:56.261307 26883 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 6 FOLLOWER]: Advancing to term 7 I0220 18:44:56.261327 27119 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 7 candidate_status { last_received { term: 5 index: 1367 } } ignore_live_leader: false dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" I0220 18:44:56.261811 27076 raft_consensus.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 6 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:56.262552 26883 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 853707498adf425fb8395cb60f41c9dc in term 7. I0220 18:44:56.262863 26690 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 853707498adf425fb8395cb60f41c9dc, a6e352f77ef54932a47d9977e15d05f1; no voters: I0220 18:44:56.263069 27076 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c751174fe9e41a0a4d3ea1dac04a50e, State: Initialized, Role: FOLLOWER I0220 18:44:56.263072 27219 raft_consensus.cc:2730] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 7 FOLLOWER]: Leader election won for term 7 I0220 18:44:56.263217 27219 raft_consensus.cc:686] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 7 LEADER]: Becoming Leader. State: Replica: 853707498adf425fb8395cb60f41c9dc, State: Running, Role: LEADER I0220 18:44:56.263532 27219 consensus_queue.cc:229] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1366, Committed index: 1366, Last appended: 5.1367, Last appended by leader: 1367, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:56.263619 27076 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1366, Last appended: 5.1367, Last appended by leader: 1367, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:56.263737 27119 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 6 FOLLOWER]: Advancing to term 7 I0220 18:44:56.265202 27119 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 853707498adf425fb8395cb60f41c9dc in term 7. I0220 18:44:56.266312 25170 catalog_manager.cc:4173] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc reported cstate change: term changed from 5 to 7, leader changed from 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132) to 853707498adf425fb8395cb60f41c9dc (127.23.135.129). New cstate: current_term: 7 leader_uuid: "853707498adf425fb8395cb60f41c9dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } health_report { overall_health: HEALTHY } } } I0220 18:44:56.267014 27076 ts_tablet_manager.cc:1142] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s W0220 18:44:56.326961 26833 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:56.327965 26833 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:56.349817 27119 raft_consensus.cc:1210] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 7 FOLLOWER]: Refusing update from remote peer 853707498adf425fb8395cb60f41c9dc: Log matching property violated. Preceding OpId in replica: term: 5 index: 1367. Preceding OpId from leader: term: 7 index: 1368. (index mismatch) I0220 18:44:56.350405 27219 consensus_queue.cc:1005] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1368, Last known committed idx: 1366, Time since last communication: 0.000s I0220 18:44:56.355315 26883 raft_consensus.cc:1210] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 7 FOLLOWER]: Refusing update from remote peer 853707498adf425fb8395cb60f41c9dc: Log matching property violated. Preceding OpId in replica: term: 5 index: 1366. Preceding OpId from leader: term: 7 index: 1368. (index mismatch) I0220 18:44:56.356791 27219 consensus_queue.cc:1005] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1368, Last known committed idx: 1366, Time since last communication: 0.000s I0220 18:44:56.360112 27239 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37906 (ReqId={client: 3bae9011ecbc40ad908b3cdb923647ae, seq_no=1363, attempt_no=63}) took 1413ms. Trace: I0220 18:44:56.360143 27239 rpcz_store.cc:270] 0220 18:44:54.947076 (+ 0us) service_pool.cc:162] Inserting onto call queue 0220 18:44:54.947134 (+ 58us) service_pool.cc:221] Handling call 0220 18:44:56.360082 (+1412948us) inbound_call.cc:162] Queueing success response Metrics: {} W0220 18:44:56.364660 26833 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 leader_uuid: "853707498adf425fb8395cb60f41c9dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:56.369501 27079 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 leader_uuid: "853707498adf425fb8395cb60f41c9dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:58.706480 25164 ts_manager.cc:267] Unset tserver state for a6e352f77ef54932a47d9977e15d05f1 from MAINTENANCE_MODE I0220 18:44:58.714850 25164 ts_manager.cc:267] Unset tserver state for 8c751174fe9e41a0a4d3ea1dac04a50e from MAINTENANCE_MODE I0220 18:44:58.743400 25164 ts_manager.cc:267] Unset tserver state for e4c29ba8de7e4745a4e497d213d63742 from MAINTENANCE_MODE I0220 18:44:58.747586 25164 ts_manager.cc:267] Unset tserver state for 853707498adf425fb8395cb60f41c9dc from MAINTENANCE_MODE I0220 18:44:59.028399 25164 ts_manager.cc:278] Set tserver state for 853707498adf425fb8395cb60f41c9dc to MAINTENANCE_MODE I0220 18:44:59.367579 26930 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:44:59.372706 27181 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:44:59.387274 26802 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:44:59.405025 25164 ts_manager.cc:278] Set tserver state for e4c29ba8de7e4745a4e497d213d63742 to MAINTENANCE_MODE I0220 18:44:59.415935 25164 ts_manager.cc:278] Set tserver state for 8c751174fe9e41a0a4d3ea1dac04a50e to MAINTENANCE_MODE I0220 18:44:59.423225 25170 ts_manager.cc:278] Set tserver state for a6e352f77ef54932a47d9977e15d05f1 to MAINTENANCE_MODE I0220 18:44:59.623474 26739 tablet_service.cc:1022] Tablet server 853707498adf425fb8395cb60f41c9dc set to quiescing I0220 18:44:59.623513 26989 tablet_service.cc:1022] Tablet server e4c29ba8de7e4745a4e497d213d63742 set to quiescing I0220 18:44:59.623598 26739 tablet_service.cc:1029] Tablet server has 1 leaders and 0 scanners I0220 18:44:59.623679 26989 tablet_service.cc:1029] Tablet server has 0 leaders and 0 scanners I0220 18:44:59.641124 27350 raft_consensus.cc:929] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: : Instructing follower 8c751174fe9e41a0a4d3ea1dac04a50e to start an election I0220 18:44:59.641268 27350 raft_consensus.cc:1017] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 7 LEADER]: Signalling peer 8c751174fe9e41a0a4d3ea1dac04a50e to start an election I0220 18:44:59.641824 27357 raft_consensus.cc:929] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: : Instructing follower a6e352f77ef54932a47d9977e15d05f1 to start an election I0220 18:44:59.641908 27357 raft_consensus.cc:1017] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 7 LEADER]: Signalling peer a6e352f77ef54932a47d9977e15d05f1 to start an election I0220 18:44:59.644556 27138 tablet_service.cc:1457] Received Run Leader Election RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" from {username='slave'} at 127.23.135.129:38733 I0220 18:44:59.644636 27138 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 7 FOLLOWER]: Starting forced leader election (received explicit request) I0220 18:44:59.644661 27138 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 7 FOLLOWER]: Advancing to term 8 I0220 18:44:59.645586 27138 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:59.646035 27119 raft_consensus.cc:1175] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 8 FOLLOWER]: Rejecting Update request from peer 853707498adf425fb8395cb60f41c9dc for earlier term 7. Current term is 8. Ops: [] I0220 18:44:59.646417 27138 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 8 election: Requested vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:44:59.648638 26883 tablet_service.cc:1457] Received Run Leader Election RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" from {username='slave'} at 127.23.135.129:39883 I0220 18:44:59.648705 26883 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 7 FOLLOWER]: Starting forced leader election (received explicit request) I0220 18:44:59.648732 26883 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 7 FOLLOWER]: Advancing to term 8 I0220 18:44:59.649662 26883 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:59.649907 26883 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 8 election: Requested vote from peers 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963), 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:44:59.651166 26753 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "a6e352f77ef54932a47d9977e15d05f1" candidate_term: 8 candidate_status { last_received { term: 7 index: 1859 } } ignore_live_leader: true dest_uuid: "853707498adf425fb8395cb60f41c9dc" I0220 18:44:59.651226 26753 raft_consensus.cc:2978] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 7 LEADER]: Stepping down as leader of term 7 I0220 18:44:59.651242 26753 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 853707498adf425fb8395cb60f41c9dc, State: Running, Role: LEADER I0220 18:44:59.651401 26753 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1859, Committed index: 1859, Last appended: 7.1861, Last appended by leader: 1861, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:44:59.651441 26753 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 7 FOLLOWER]: Advancing to term 8 I0220 18:44:59.651571 27138 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "a6e352f77ef54932a47d9977e15d05f1" candidate_term: 8 candidate_status { last_received { term: 7 index: 1859 } } ignore_live_leader: true dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" I0220 18:44:59.651715 27138 raft_consensus.cc:2339] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a6e352f77ef54932a47d9977e15d05f1 in current term 8: Already voted for candidate 8c751174fe9e41a0a4d3ea1dac04a50e in this term. I0220 18:44:59.652325 26753 raft_consensus.cc:2356] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a6e352f77ef54932a47d9977e15d05f1 for term 8 because replica has last-logged OpId of term: 7 index: 1861, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 1859. W0220 18:44:59.652597 26707 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:59.652976 26815 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a6e352f77ef54932a47d9977e15d05f1; no voters: 853707498adf425fb8395cb60f41c9dc, 8c751174fe9e41a0a4d3ea1dac04a50e I0220 18:44:59.653240 26883 raft_consensus.cc:1175] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 8 FOLLOWER]: Rejecting Update request from peer 853707498adf425fb8395cb60f41c9dc for earlier term 7. Current term is 8. Ops: [] I0220 18:44:59.654001 26753 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 8 candidate_status { last_received { term: 7 index: 1859 } } ignore_live_leader: true dest_uuid: "853707498adf425fb8395cb60f41c9dc" I0220 18:44:59.654081 26753 raft_consensus.cc:2356] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c751174fe9e41a0a4d3ea1dac04a50e for term 8 because replica has last-logged OpId of term: 7 index: 1861, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 1859. I0220 18:44:59.659384 27420 raft_consensus.cc:2675] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority W0220 18:44:59.659689 26832 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:59.663355 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:59.666484 26883 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 8 candidate_status { last_received { term: 7 index: 1859 } } ignore_live_leader: true dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" I0220 18:44:59.666574 26883 raft_consensus.cc:2339] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c751174fe9e41a0a4d3ea1dac04a50e in current term 8: Already voted for candidate a6e352f77ef54932a47d9977e15d05f1 in this term. I0220 18:44:59.667001 27069 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c751174fe9e41a0a4d3ea1dac04a50e; no voters: 853707498adf425fb8395cb60f41c9dc, a6e352f77ef54932a47d9977e15d05f1 I0220 18:44:59.668915 27421 raft_consensus.cc:2675] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority W0220 18:44:59.669288 26707 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:59.675130 26833 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:59.682772 27056 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... W0220 18:44:59.683612 27079 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:59.696390 26707 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:59.707075 26833 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:59.717490 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:59.734030 26707 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:59.745601 26834 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:59.766793 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:59.781692 26849 tablet_service.cc:1022] Tablet server a6e352f77ef54932a47d9977e15d05f1 set to quiescing I0220 18:44:59.781805 26849 tablet_service.cc:1029] Tablet server has 0 leaders and 1 scanners W0220 18:44:59.785392 26707 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:44:59.795028 27113 tablet_service.cc:1022] Tablet server 8c751174fe9e41a0a4d3ea1dac04a50e set to quiescing I0220 18:44:59.795135 27113 tablet_service.cc:1029] Tablet server has 0 leaders and 2 scanners W0220 18:44:59.801976 26833 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:59.817919 27098 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:59.839835 26707 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:59.858177 26833 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:59.877807 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:59.902884 26707 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:59.926610 26833 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:59.950801 27079 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:44:59.976253 26707 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.001876 26833 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.019506 27350 raft_consensus.cc:659] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: failed to trigger leader election: Illegal state: leader elections are disabled W0220 18:45:00.026160 27420 raft_consensus.cc:659] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: failed to trigger leader election: Illegal state: leader elections are disabled W0220 18:45:00.027623 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.054713 26707 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.084324 26834 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.084872 27421 raft_consensus.cc:659] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: failed to trigger leader election: Illegal state: leader elections are disabled W0220 18:45:00.117125 27079 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.150537 26707 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.184348 26833 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.217902 27098 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.252754 26707 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.289084 26833 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.324167 27098 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.360823 26707 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.397087 26833 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.436794 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.479882 26707 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.519171 26833 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.563094 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.607023 26707 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.652885 26833 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.698802 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:00.715382 26739 tablet_service.cc:1022] Tablet server 853707498adf425fb8395cb60f41c9dc set to quiescing I0220 18:45:00.715481 26739 tablet_service.cc:1029] Tablet server has 0 leaders and 0 scanners W0220 18:45:00.748327 26707 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.795344 26833 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.841611 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:00.888859 26849 tablet_service.cc:1022] Tablet server a6e352f77ef54932a47d9977e15d05f1 set to quiescing I0220 18:45:00.889009 26849 tablet_service.cc:1029] Tablet server has 0 leaders and 0 scanners W0220 18:45:00.890605 26707 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:37906: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:00.891100 27113 tablet_service.cc:1022] Tablet server 8c751174fe9e41a0a4d3ea1dac04a50e set to quiescing I0220 18:45:00.891209 27113 tablet_service.cc:1029] Tablet server has 0 leaders and 0 scanners I0220 18:45:00.897210 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 26679 W0220 18:45:00.902443 25721 connection.cc:658] client connection to 127.23.135.129:34551 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) W0220 18:45:00.902496 25721 meta_cache.cc:234] Tablet 7e19d7c23cc941e3a7751fe2478481ec: Replica 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) has failed: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) I0220 18:45:00.902601 25752 meta_cache.cc:1038] Marking tablet server 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) as failed. I0220 18:45:00.902978 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.23.135.129:34551 --local_ip_for_outbound_sockets=127.23.135.129 --webserver_port=33607 --webserver_interface=127.23.135.129 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} W0220 18:45:00.905743 26833 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.911257 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.912240 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.920203 26833 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.925323 26833 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.931944 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.940145 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.941092 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.950924 26833 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:45:00.951232 27474 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:45:00.951436 27474 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:45:00.952514 27474 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:45:00.952589 27474 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.129 W0220 18:45:00.952744 26834 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:00.954025 27474 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.129:34551 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/info.pb --webserver_interface=127.23.135.129 --webserver_port=33607 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.27474 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.129 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:45:00.954502 27474 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:45:00.955144 27474 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:45:00.960135 27474 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:45:00.960883 27474 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:45:00.962038 27474 hybrid_clock.cc:552] HybridClock initialized: now 1582224300962010 us; error 24 us; skew 500 ppm I0220 18:45:00.962549 27474 fs_manager.cc:259] Metadata directory not provided I0220 18:45:00.962580 27474 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory W0220 18:45:00.966351 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:00.967489 27474 fs_manager.cc:410] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s I0220 18:45:00.970454 27474 fs_manager.cc:430] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0220 18:45:00.970846 27474 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal uuid: "853707498adf425fb8395cb60f41c9dc" format_stamp: "Formatted at 2020-02-20 18:44:40 on dist-test-slave-dist-test-slave-sbf6" I0220 18:45:00.970927 27474 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0220 18:45:00.974719 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:00.985992 26834 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:00.987938 27474 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:45:00.988799 27474 ts_tablet_manager.cc:383] Loading tablet metadata (0/1 complete) W0220 18:45:00.988930 26834 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:00.991091 27474 ts_tablet_manager.cc:395] Loaded tablet metadata (1 total tablets, 1 live tablets) I0220 18:45:00.991124 27474 ts_tablet_manager.cc:401] Registering tablets (0/1 complete) I0220 18:45:00.992203 27474 ts_tablet_manager.cc:415] Registered 1 tablets I0220 18:45:00.992547 27492 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Bootstrap starting. W0220 18:45:00.993243 26834 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34632: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:01.001374 27474 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.129:34551 I0220 18:45:01.002032 27474 webserver.cc:382] Webserver started at http://127.23.135.129:33607/ using document root and password file I0220 18:45:01.002537 27474 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/info.pb W0220 18:45:01.005276 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:01.009183 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 27474 I0220 18:45:01.009255 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 26807 I0220 18:45:01.012527 27597 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:45:01.012712 27597 heartbeater.cc:406] Registering TS with master... I0220 18:45:01.013068 27597 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:01.014381 25170 ts_manager.cc:192] Re-registered known tserver with Master: 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:45:01.015530 25170 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.129:41809 W0220 18:45:01.015733 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:01.016590 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.23.135.130:36293 --local_ip_for_outbound_sockets=127.23.135.130 --webserver_port=39793 --webserver_interface=127.23.135.130 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} W0220 18:45:01.051806 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:01.088519 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:45:01.109772 27601 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:45:01.109911 27601 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:45:01.110797 27601 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:45:01.110850 27601 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.130 I0220 18:45:01.116375 27601 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.130:36293 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/info.pb --webserver_interface=127.23.135.130 --webserver_port=39793 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.27601 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.130 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:45:01.116925 27601 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:45:01.117668 27601 file_cache.cc:476] Constructed file cache file cache with capacity 419430 W0220 18:45:01.126643 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:01.126698 27601 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:45:01.127521 27601 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize W0220 18:45:01.128799 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:01.142997 27601 hybrid_clock.cc:552] HybridClock initialized: now 1582224301142952 us; error 37 us; skew 500 ppm I0220 18:45:01.143604 27601 fs_manager.cc:259] Metadata directory not provided I0220 18:45:01.143677 27601 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:45:01.152446 27601 fs_manager.cc:410] Time spent opening directory manager: real 0.008s user 0.004s sys 0.000s I0220 18:45:01.155863 27601 fs_manager.cc:430] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0220 18:45:01.156297 27601 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal uuid: "a6e352f77ef54932a47d9977e15d05f1" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:45:01.156388 27601 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:45:01.171734 27601 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:45:01.172549 27601 ts_tablet_manager.cc:383] Loading tablet metadata (0/1 complete) W0220 18:45:01.174072 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:01.175076 27601 ts_tablet_manager.cc:395] Loaded tablet metadata (1 total tablets, 1 live tablets) I0220 18:45:01.175101 27601 ts_tablet_manager.cc:401] Registering tablets (0/1 complete) I0220 18:45:01.176162 27601 ts_tablet_manager.cc:415] Registered 1 tablets I0220 18:45:01.180231 27619 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Bootstrap starting. I0220 18:45:01.183054 27601 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.130:36293 I0220 18:45:01.183991 27601 webserver.cc:382] Webserver started at http://127.23.135.130:39793/ using document root and password file I0220 18:45:01.184511 27601 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/info.pb I0220 18:45:01.193632 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 27601 I0220 18:45:01.193719 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 26933 W0220 18:45:01.196228 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:01.200708 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.23.135.131:45349 --local_ip_for_outbound_sockets=127.23.135.131 --webserver_port=35617 --webserver_interface=127.23.135.131 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} I0220 18:45:01.208555 27724 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:45:01.208712 27724 heartbeater.cc:406] Registering TS with master... I0220 18:45:01.209195 27724 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:01.210567 25170 ts_manager.cc:192] Re-registered known tserver with Master: a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293) I0220 18:45:01.212064 25170 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.130:52371 I0220 18:45:01.223518 27492 log.cc:828] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Log is configured to *not* fsync() on all Append() calls W0220 18:45:01.250952 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:45:01.258716 27727 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:45:01.258859 27727 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:45:01.259753 27727 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:45:01.259821 27727 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.131 I0220 18:45:01.261270 27727 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.131:45349 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/info.pb --webserver_interface=127.23.135.131 --webserver_port=35617 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.27727 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.131 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:45:01.261724 27727 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:45:01.262341 27727 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:45:01.267072 27727 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:45:01.267815 27727 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize W0220 18:45:01.270381 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:01.275142 27727 hybrid_clock.cc:552] HybridClock initialized: now 1582224301275109 us; error 33 us; skew 500 ppm I0220 18:45:01.275748 27727 fs_manager.cc:259] Metadata directory not provided I0220 18:45:01.275784 27727 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory W0220 18:45:01.276703 27097 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39712: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:01.284466 27727 fs_manager.cc:410] Time spent opening directory manager: real 0.007s user 0.003s sys 0.001s I0220 18:45:01.287487 27727 fs_manager.cc:430] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0220 18:45:01.287952 27727 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal uuid: "e4c29ba8de7e4745a4e497d213d63742" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:45:01.288066 27727 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:45:01.309562 27727 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:45:01.310351 27727 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0220 18:45:01.310379 27727 ts_tablet_manager.cc:415] Registered 0 tablets I0220 18:45:01.316478 27727 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.131:45349 I0220 18:45:01.317253 27727 webserver.cc:382] Webserver started at http://127.23.135.131:35617/ using document root and password file I0220 18:45:01.317701 27727 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/info.pb I0220 18:45:01.327337 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 27727 I0220 18:45:01.327419 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 27059 I0220 18:45:01.332988 27850 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:45:01.333120 27850 heartbeater.cc:406] Registering TS with master... I0220 18:45:01.333514 27850 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:01.334517 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data --block_manager=log --rpc_bind_addresses=127.23.135.132:33963 --local_ip_for_outbound_sockets=127.23.135.132 --webserver_port=44317 --webserver_interface=127.23.135.132 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} I0220 18:45:01.334653 25170 ts_manager.cc:192] Re-registered known tserver with Master: e4c29ba8de7e4745a4e497d213d63742 (127.23.135.131:45349) I0220 18:45:01.335466 25170 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.131:58725 WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:45:01.380190 27853 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:45:01.380340 27853 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:45:01.381248 27853 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:45:01.381316 27853 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.132 I0220 18:45:01.382603 27853 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.132:33963 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/info.pb --webserver_interface=127.23.135.132 --webserver_port=44317 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.27853 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.132 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:45:01.383066 27853 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:45:01.383785 27853 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:45:01.388624 27853 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:45:01.389484 27853 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:45:01.396802 27853 hybrid_clock.cc:552] HybridClock initialized: now 1582224301396761 us; error 38 us; skew 500 ppm I0220 18:45:01.397372 27853 fs_manager.cc:259] Metadata directory not provided I0220 18:45:01.397411 27853 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:45:01.402815 27853 fs_manager.cc:410] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s I0220 18:45:01.406042 27853 fs_manager.cc:430] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0220 18:45:01.406481 27853 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:45:01.406580 27853 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:45:01.408586 27619 log.cc:828] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Log is configured to *not* fsync() on all Append() calls I0220 18:45:01.420300 27853 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:45:01.421177 27853 ts_tablet_manager.cc:383] Loading tablet metadata (0/1 complete) I0220 18:45:01.423348 27853 ts_tablet_manager.cc:395] Loaded tablet metadata (1 total tablets, 1 live tablets) I0220 18:45:01.423384 27853 ts_tablet_manager.cc:401] Registering tablets (0/1 complete) I0220 18:45:01.424780 27853 ts_tablet_manager.cc:415] Registered 1 tablets I0220 18:45:01.427999 27871 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Bootstrap starting. I0220 18:45:01.437544 27853 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.132:33963 I0220 18:45:01.438269 27853 webserver.cc:382] Webserver started at http://127.23.135.132:44317/ using document root and password file I0220 18:45:01.438799 27853 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/info.pb I0220 18:45:01.441442 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 27853 I0220 18:45:01.455457 27976 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:45:01.455590 27976 heartbeater.cc:406] Registering TS with master... I0220 18:45:01.456050 27976 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:01.457418 25170 ts_manager.cc:192] Re-registered known tserver with Master: 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) I0220 18:45:01.458497 25170 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.132:36675 I0220 18:45:01.599949 27871 log.cc:828] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Log is configured to *not* fsync() on all Append() calls I0220 18:45:02.017781 27597 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:45:02.213806 27724 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:45:02.337370 27850 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:45:02.461642 27976 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:45:04.931486 27871 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1859 overwritten=0 applied=1859 ignored=0} inserts{seen=92750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0220 18:45:04.932013 27871 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Bootstrap complete. I0220 18:45:04.933531 27871 ts_tablet_manager.cc:1113] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Time spent bootstrapping tablet: real 3.506s user 3.374s sys 0.100s I0220 18:45:04.936910 27871 raft_consensus.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 8 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:04.936986 27871 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c751174fe9e41a0a4d3ea1dac04a50e, State: Initialized, Role: FOLLOWER I0220 18:45:04.937423 27871 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1859, Last appended: 7.1859, Last appended by leader: 1859, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:04.938419 27871 ts_tablet_manager.cc:1142] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s W0220 18:45:05.065285 27882 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:05.070262 27882 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:05.071254 27882 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:05.222481 27492 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=1861 overwritten=0 applied=1859 ignored=0} inserts{seen=92750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates I0220 18:45:05.222877 27492 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Bootstrap complete. I0220 18:45:05.224112 27492 ts_tablet_manager.cc:1113] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Time spent bootstrapping tablet: real 4.232s user 3.926s sys 0.104s I0220 18:45:05.226521 27492 raft_consensus.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 8 FOLLOWER]: Replica starting. Triggering 2 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:05.227602 27492 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853707498adf425fb8395cb60f41c9dc, State: Initialized, Role: FOLLOWER I0220 18:45:05.227978 27492 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1859, Last appended: 7.1861, Last appended by leader: 1861, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:05.228418 27492 ts_tablet_manager.cc:1142] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s I0220 18:45:05.244228 28013 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:45:05.244401 28013 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:05.245434 28013 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:45:05.250458 27553 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 9 candidate_status { last_received { term: 7 index: 1859 } } ignore_live_leader: false dest_uuid: "853707498adf425fb8395cb60f41c9dc" is_pre_election: true I0220 18:45:05.250622 27553 raft_consensus.cc:2356] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c751174fe9e41a0a4d3ea1dac04a50e for term 9 because replica has last-logged OpId of term: 7 index: 1861, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 1859. I0220 18:45:05.251696 27680 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 9 candidate_status { last_received { term: 7 index: 1859 } } ignore_live_leader: false dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" is_pre_election: true W0220 18:45:05.252120 27864 leader_election.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 9 pre-election: Tablet error from VoteRequest() call to peer a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293): Illegal state: must be running to vote when last-logged opid is not known I0220 18:45:05.252183 27864 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c751174fe9e41a0a4d3ea1dac04a50e; no voters: 853707498adf425fb8395cb60f41c9dc, a6e352f77ef54932a47d9977e15d05f1 I0220 18:45:05.252436 28013 raft_consensus.cc:2675] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority W0220 18:45:05.347476 27882 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:05.354705 27882 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:05.376322 27882 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:05.484078 27514 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38004: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:05.507228 28015 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:45:05.507423 28015 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:05.508368 28015 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) I0220 18:45:05.513407 27931 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 9 candidate_status { last_received { term: 7 index: 1861 } } ignore_live_leader: false dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" is_pre_election: true I0220 18:45:05.513552 27931 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 853707498adf425fb8395cb60f41c9dc in term 8. I0220 18:45:05.513633 27680 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 9 candidate_status { last_received { term: 7 index: 1861 } } ignore_live_leader: false dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" is_pre_election: true I0220 18:45:05.513937 27483 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 853707498adf425fb8395cb60f41c9dc, 8c751174fe9e41a0a4d3ea1dac04a50e; no voters: W0220 18:45:05.514061 27485 leader_election.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 9 pre-election: Tablet error from VoteRequest() call to peer a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293): Illegal state: must be running to vote when last-logged opid is not known I0220 18:45:05.514333 28015 raft_consensus.cc:2730] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 8 FOLLOWER]: Leader pre-election won for term 9 I0220 18:45:05.514370 28015 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0220 18:45:05.514387 28015 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 8 FOLLOWER]: Advancing to term 9 I0220 18:45:05.515990 28015 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:05.516273 28015 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 9 election: Requested vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) I0220 18:45:05.516610 27931 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 9 candidate_status { last_received { term: 7 index: 1861 } } ignore_live_leader: false dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" I0220 18:45:05.516685 27931 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 8 FOLLOWER]: Advancing to term 9 I0220 18:45:05.517999 27931 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 853707498adf425fb8395cb60f41c9dc in term 9. I0220 18:45:05.518254 27680 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 9 candidate_status { last_received { term: 7 index: 1861 } } ignore_live_leader: false dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" I0220 18:45:05.518298 27483 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 853707498adf425fb8395cb60f41c9dc, 8c751174fe9e41a0a4d3ea1dac04a50e; no voters: W0220 18:45:05.518538 27485 leader_election.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 9 election: Tablet error from VoteRequest() call to peer a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293): Illegal state: must be running to vote when last-logged opid is not known I0220 18:45:05.518546 28015 raft_consensus.cc:2730] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 9 FOLLOWER]: Leader election won for term 9 I0220 18:45:05.518728 28015 raft_consensus.cc:686] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 9 LEADER]: Becoming Leader. State: Replica: 853707498adf425fb8395cb60f41c9dc, State: Running, Role: LEADER I0220 18:45:05.518996 28015 consensus_queue.cc:229] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1859, Committed index: 1859, Last appended: 7.1861, Last appended by leader: 1861, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:05.521319 25170 catalog_manager.cc:4173] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "853707498adf425fb8395cb60f41c9dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } health_report { overall_health: HEALTHY } } } I0220 18:45:05.603212 27931 raft_consensus.cc:1210] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 9 FOLLOWER]: Refusing update from remote peer 853707498adf425fb8395cb60f41c9dc: Log matching property violated. Preceding OpId in replica: term: 7 index: 1859. Preceding OpId from leader: term: 9 index: 1862. (index mismatch) I0220 18:45:05.603953 28015 consensus_queue.cc:1005] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1862, Last known committed idx: 1859, Time since last communication: 0.000s W0220 18:45:05.609870 27485 consensus_peers.cc:472] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc -> Peer a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293): Couldn't send request to peer a6e352f77ef54932a47d9977e15d05f1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry. W0220 18:45:05.619065 27882 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 9 leader_uuid: "853707498adf425fb8395cb60f41c9dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:05.621294 27882 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 9 leader_uuid: "853707498adf425fb8395cb60f41c9dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:05.642572 27619 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=1859 overwritten=0 applied=1859 ignored=0} inserts{seen=92750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0220 18:45:05.643091 27619 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Bootstrap complete. I0220 18:45:05.644714 27619 ts_tablet_manager.cc:1113] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Time spent bootstrapping tablet: real 4.465s user 4.279s sys 0.103s I0220 18:45:05.647361 27619 raft_consensus.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:05.647475 27619 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e352f77ef54932a47d9977e15d05f1, State: Initialized, Role: FOLLOWER I0220 18:45:05.648028 27619 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1859, Last appended: 7.1859, Last appended by leader: 1859, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:05.649327 27619 ts_tablet_manager.cc:1142] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s W0220 18:45:05.701035 27882 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 9 leader_uuid: "853707498adf425fb8395cb60f41c9dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:05.703847 27882 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 9 leader_uuid: "853707498adf425fb8395cb60f41c9dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:05.705371 27882 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 9 leader_uuid: "853707498adf425fb8395cb60f41c9dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:05.707338 27640 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34736: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:05.708539 27640 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34736: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:05.728139 27680 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 8 FOLLOWER]: Advancing to term 9 I0220 18:45:05.729713 27680 raft_consensus.cc:1210] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 9 FOLLOWER]: Refusing update from remote peer 853707498adf425fb8395cb60f41c9dc: Log matching property violated. Preceding OpId in replica: term: 7 index: 1859. Preceding OpId from leader: term: 7 index: 1861. (index mismatch) I0220 18:45:05.730497 28035 consensus_queue.cc:1007] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1862, Last known committed idx: 1859, Time since last communication: 0.000s I0220 18:45:05.782666 28037 mvcc.cc:203] Tried to move safe_time back from 6480790756270743552 to 6480790755407126528. Current Snapshot: MvccSnapshot[committed={T|T < 6480790756270743552 or (T in {6480790756270743552})}] W0220 18:45:05.811591 27640 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34736: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 9 leader_uuid: "853707498adf425fb8395cb60f41c9dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:07.271739 25164 ts_manager.cc:267] Unset tserver state for 8c751174fe9e41a0a4d3ea1dac04a50e from MAINTENANCE_MODE I0220 18:45:07.347213 27850 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:07.411243 25164 ts_manager.cc:267] Unset tserver state for a6e352f77ef54932a47d9977e15d05f1 from MAINTENANCE_MODE I0220 18:45:07.413246 25164 ts_manager.cc:267] Unset tserver state for 853707498adf425fb8395cb60f41c9dc from MAINTENANCE_MODE I0220 18:45:07.457325 25164 ts_manager.cc:267] Unset tserver state for e4c29ba8de7e4745a4e497d213d63742 from MAINTENANCE_MODE I0220 18:45:07.614573 27976 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:07.742410 27724 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:07.745959 27597 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:07.929229 25164 ts_manager.cc:278] Set tserver state for e4c29ba8de7e4745a4e497d213d63742 to MAINTENANCE_MODE I0220 18:45:07.937328 25164 ts_manager.cc:278] Set tserver state for a6e352f77ef54932a47d9977e15d05f1 to MAINTENANCE_MODE I0220 18:45:07.939213 25170 ts_manager.cc:278] Set tserver state for 853707498adf425fb8395cb60f41c9dc to MAINTENANCE_MODE I0220 18:45:07.941334 25164 ts_manager.cc:278] Set tserver state for 8c751174fe9e41a0a4d3ea1dac04a50e to MAINTENANCE_MODE I0220 18:45:08.113895 27897 tablet_service.cc:1022] Tablet server 8c751174fe9e41a0a4d3ea1dac04a50e set to quiescing I0220 18:45:08.114066 27897 tablet_service.cc:1029] Tablet server has 0 leaders and 1 scanners I0220 18:45:08.132791 27787 tablet_service.cc:1022] Tablet server e4c29ba8de7e4745a4e497d213d63742 set to quiescing I0220 18:45:08.132915 27787 tablet_service.cc:1029] Tablet server has 0 leaders and 0 scanners I0220 18:45:08.264387 27534 tablet_service.cc:1022] Tablet server 853707498adf425fb8395cb60f41c9dc set to quiescing I0220 18:45:08.264544 27534 tablet_service.cc:1029] Tablet server has 1 leaders and 2 scanners I0220 18:45:08.265722 28042 raft_consensus.cc:929] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: : Instructing follower a6e352f77ef54932a47d9977e15d05f1 to start an election I0220 18:45:08.265856 28042 raft_consensus.cc:1017] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 9 LEADER]: Signalling peer a6e352f77ef54932a47d9977e15d05f1 to start an election I0220 18:45:08.266675 27681 tablet_service.cc:1457] Received Run Leader Election RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" from {username='slave'} at 127.23.135.129:40291 I0220 18:45:08.266757 27681 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 9 FOLLOWER]: Starting forced leader election (received explicit request) I0220 18:45:08.266813 27681 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 9 FOLLOWER]: Advancing to term 10 I0220 18:45:08.267911 27681 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:08.268777 27681 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 10 election: Requested vote from peers 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963), 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:45:08.274474 27650 tablet_service.cc:1022] Tablet server a6e352f77ef54932a47d9977e15d05f1 set to quiescing I0220 18:45:08.274585 27650 tablet_service.cc:1029] Tablet server has 0 leaders and 0 scanners I0220 18:45:08.274713 27933 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "a6e352f77ef54932a47d9977e15d05f1" candidate_term: 10 candidate_status { last_received { term: 9 index: 2268 } } ignore_live_leader: true dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" I0220 18:45:08.274770 27933 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 9 FOLLOWER]: Advancing to term 10 I0220 18:45:08.275269 27681 raft_consensus.cc:1175] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 10 FOLLOWER]: Rejecting Update request from peer 853707498adf425fb8395cb60f41c9dc for earlier term 9. Current term is 10. Ops: [] I0220 18:45:08.275811 28035 consensus_queue.cc:1016] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 }, Status: INVALID_TERM, Last received: 9.2268, Next index: 2269, Last known committed idx: 2268, Time since last communication: 0.000s I0220 18:45:08.275816 27933 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e352f77ef54932a47d9977e15d05f1 in term 10. I0220 18:45:08.276235 28035 raft_consensus.cc:2978] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 9 LEADER]: Stepping down as leader of term 9 I0220 18:45:08.276259 28035 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 853707498adf425fb8395cb60f41c9dc, State: Running, Role: LEADER I0220 18:45:08.276398 28035 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2268, Committed index: 2268, Last appended: 9.2270, Last appended by leader: 2270, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:08.276540 28035 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 9 FOLLOWER]: Advancing to term 10 I0220 18:45:08.277321 27933 raft_consensus.cc:1175] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 10 FOLLOWER]: Rejecting Update request from peer 853707498adf425fb8395cb60f41c9dc for earlier term 9. Current term is 10. Ops: [] I0220 18:45:08.277706 27610 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c751174fe9e41a0a4d3ea1dac04a50e, a6e352f77ef54932a47d9977e15d05f1; no voters: I0220 18:45:08.278187 27553 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "a6e352f77ef54932a47d9977e15d05f1" candidate_term: 10 candidate_status { last_received { term: 9 index: 2268 } } ignore_live_leader: true dest_uuid: "853707498adf425fb8395cb60f41c9dc" I0220 18:45:08.278271 27553 raft_consensus.cc:2356] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a6e352f77ef54932a47d9977e15d05f1 for term 10 because replica has last-logged OpId of term: 9 index: 2270, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 2268. I0220 18:45:08.279228 28225 raft_consensus.cc:2730] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 10 FOLLOWER]: Leader election won for term 10 I0220 18:45:08.279419 28225 raft_consensus.cc:686] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 10 LEADER]: Becoming Leader. State: Replica: a6e352f77ef54932a47d9977e15d05f1, State: Running, Role: LEADER I0220 18:45:08.281134 28225 consensus_queue.cc:229] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2268, Committed index: 2268, Last appended: 9.2268, Last appended by leader: 2268, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } W0220 18:45:08.284065 27511 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38004: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:08.285465 25164 catalog_manager.cc:4173] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 reported cstate change: term changed from 9 to 10, leader changed from 853707498adf425fb8395cb60f41c9dc (127.23.135.129) to a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130). New cstate: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } health_report { overall_health: UNKNOWN } } } I0220 18:45:08.292937 27553 raft_consensus.cc:1210] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 10 FOLLOWER]: Refusing update from remote peer a6e352f77ef54932a47d9977e15d05f1: Log matching property violated. Preceding OpId in replica: term: 9 index: 2270. Preceding OpId from leader: term: 10 index: 2270. (term mismatch) I0220 18:45:08.292977 27553 pending_rounds.cc:78] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Aborting all transactions after (but not including) 2269 I0220 18:45:08.292996 27553 pending_rounds.cc:99] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Aborting uncommitted WRITE_OP operation due to leader change: 9.2270 W0220 18:45:08.293309 27553 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38004: Aborted: Transaction aborted by new leader I0220 18:45:08.293679 27933 raft_consensus.cc:1210] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 10 FOLLOWER]: Refusing update from remote peer a6e352f77ef54932a47d9977e15d05f1: Log matching property violated. Preceding OpId in replica: term: 9 index: 2268. Preceding OpId from leader: term: 10 index: 2270. (index mismatch) I0220 18:45:08.294212 28225 consensus_queue.cc:1005] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2269, Last known committed idx: 2268, Time since last communication: 0.000s I0220 18:45:08.294312 28225 consensus_queue.cc:1196] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [LEADER]: Peer 853707498adf425fb8395cb60f41c9dc log is divergent from this leader: its last log entry 9.2269 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2268 I0220 18:45:08.294625 28225 consensus_queue.cc:1005] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2269, Last known committed idx: 2268, Time since last communication: 0.000s I0220 18:45:08.294992 27553 pending_rounds.cc:78] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Aborting all transactions after (but not including) 2268 I0220 18:45:08.295022 27553 pending_rounds.cc:99] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Aborting uncommitted WRITE_OP operation due to leader change: 9.2269 W0220 18:45:08.295236 27553 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38004: Aborted: Transaction aborted by new leader I0220 18:45:08.299942 28014 mvcc.cc:203] Tried to move safe_time back from 6480790766763048960 to 6480790766727483392. Current Snapshot: MvccSnapshot[committed={T|T < 6480790766763048960 or (T in {6480790766763048960})}] I0220 18:45:08.308480 28029 mvcc.cc:203] Tried to move safe_time back from 6480790766763048960 to 6480790766727483392. Current Snapshot: MvccSnapshot[committed={T|T < 6480790766763048960 or (T in {6480790766763048960})}] I0220 18:45:08.349776 27850 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:09.368165 27897 tablet_service.cc:1022] Tablet server 8c751174fe9e41a0a4d3ea1dac04a50e set to quiescing I0220 18:45:09.368269 27897 tablet_service.cc:1029] Tablet server has 0 leaders and 1 scanners I0220 18:45:09.430677 27534 tablet_service.cc:1022] Tablet server 853707498adf425fb8395cb60f41c9dc set to quiescing I0220 18:45:09.430828 27534 tablet_service.cc:1029] Tablet server has 0 leaders and 2 scanners I0220 18:45:10.583218 27897 tablet_service.cc:1022] Tablet server 8c751174fe9e41a0a4d3ea1dac04a50e set to quiescing I0220 18:45:10.583328 27897 tablet_service.cc:1029] Tablet server has 0 leaders and 0 scanners I0220 18:45:10.633626 27534 tablet_service.cc:1022] Tablet server 853707498adf425fb8395cb60f41c9dc set to quiescing I0220 18:45:10.633734 27534 tablet_service.cc:1029] Tablet server has 0 leaders and 0 scanners I0220 18:45:10.641288 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 27474 W0220 18:45:10.651926 27610 consensus_peers.cc:472] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 -> Peer 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551): Couldn't send request to peer 853707498adf425fb8395cb60f41c9dc. Status: Network error: recv got EOF from 127.23.135.129:34551 (error 108). This is attempt 1: this message will repeat every 5th retry. I0220 18:45:10.652182 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.23.135.129:34551 --local_ip_for_outbound_sockets=127.23.135.129 --webserver_port=33607 --webserver_interface=127.23.135.129 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:45:10.720563 28280 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:45:10.720741 28280 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:45:10.721714 28280 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:45:10.721786 28280 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.129 I0220 18:45:10.723474 28280 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.129:34551 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/info.pb --webserver_interface=127.23.135.129 --webserver_port=33607 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.28280 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.129 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:45:10.724165 28280 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:45:10.724932 28280 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:45:10.731266 28280 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:45:10.732210 28280 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:45:10.735414 28280 hybrid_clock.cc:552] HybridClock initialized: now 1582224310735377 us; error 33 us; skew 500 ppm I0220 18:45:10.738193 28280 fs_manager.cc:259] Metadata directory not provided I0220 18:45:10.738238 28280 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:45:10.743494 28280 fs_manager.cc:410] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s I0220 18:45:10.753118 28280 fs_manager.cc:430] Time spent opening block manager: real 0.007s user 0.000s sys 0.002s I0220 18:45:10.753692 28280 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal uuid: "853707498adf425fb8395cb60f41c9dc" format_stamp: "Formatted at 2020-02-20 18:44:40 on dist-test-slave-dist-test-slave-sbf6" I0220 18:45:10.753801 28280 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:45:10.786705 28280 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:45:10.787766 28280 ts_tablet_manager.cc:383] Loading tablet metadata (0/1 complete) I0220 18:45:10.793773 28280 ts_tablet_manager.cc:395] Loaded tablet metadata (1 total tablets, 1 live tablets) I0220 18:45:10.793884 28280 ts_tablet_manager.cc:401] Registering tablets (0/1 complete) I0220 18:45:10.795279 28280 ts_tablet_manager.cc:415] Registered 1 tablets I0220 18:45:10.795964 28298 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Bootstrap starting. I0220 18:45:10.823787 28280 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.129:34551 I0220 18:45:10.824702 28280 webserver.cc:382] Webserver started at http://127.23.135.129:33607/ using document root and password file I0220 18:45:10.825320 28280 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/info.pb I0220 18:45:10.831553 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 28280 I0220 18:45:10.831622 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 27601 W0220 18:45:10.837663 27864 connection.cc:658] server connection from 127.23.135.130:60519 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) W0220 18:45:10.838086 28407 negotiation.cc:313] Failed RPC negotiation. Trace: 0220 18:45:10.829304 (+ 0us) reactor.cc:603] Submitting negotiation task for server connection from 127.23.135.130:44239 0220 18:45:10.831394 (+ 2090us) server_negotiation.cc:196] Beginning negotiation 0220 18:45:10.831401 (+ 7us) server_negotiation.cc:385] Waiting for connection header 0220 18:45:10.831424 (+ 23us) server_negotiation.cc:393] Connection header received 0220 18:45:10.831553 (+ 129us) server_negotiation.cc:349] Received NEGOTIATE NegotiatePB request 0220 18:45:10.831558 (+ 5us) server_negotiation.cc:445] Received NEGOTIATE request from client 0220 18:45:10.832138 (+ 580us) server_negotiation.cc:361] Sending NEGOTIATE NegotiatePB response 0220 18:45:10.832254 (+ 116us) server_negotiation.cc:217] Negotiated authn=SASL 0220 18:45:10.837730 (+ 5476us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.135.130:44239: BlockingRecv error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) Metrics: {"server-negotiator.queue_time_us":1989,"thread_start_us":82,"threads_started":1} W0220 18:45:10.838599 25723 connection.cc:658] client connection to 127.23.135.130:36293 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) W0220 18:45:10.838655 25723 meta_cache.cc:234] Tablet 7e19d7c23cc941e3a7751fe2478481ec: Replica a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293) has failed: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) I0220 18:45:10.839284 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.23.135.130:36293 --local_ip_for_outbound_sockets=127.23.135.130 --webserver_port=39793 --webserver_interface=127.23.135.130 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} W0220 18:45:10.842383 27893 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:10.842556 27875 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:10.844727 28403 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:45:10.844853 28403 heartbeater.cc:406] Registering TS with master... W0220 18:45:10.844878 27875 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:10.845244 28403 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:10.846369 25164 ts_manager.cc:192] Re-registered known tserver with Master: 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:45:10.847205 25164 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.129:34835 W0220 18:45:10.872542 27875 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:10.872596 27893 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:10.875428 27875 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:10.899046 27875 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:10.901183 27875 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:10.903280 27875 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:45:10.941401 28408 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:45:10.941562 28408 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:45:10.941874 27893 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:10.941934 27875 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:10.944655 28408 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:45:10.944736 28408 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.130 W0220 18:45:10.944881 27875 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:10.946202 28408 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.130:36293 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/info.pb --webserver_interface=127.23.135.130 --webserver_port=39793 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.28408 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.130 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:45:10.946687 28408 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:45:10.947387 28408 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:45:10.968895 28408 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:45:10.969826 28408 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:45:10.971014 28408 hybrid_clock.cc:552] HybridClock initialized: now 1582224310970982 us; error 25 us; skew 500 ppm I0220 18:45:10.971741 28408 fs_manager.cc:259] Metadata directory not provided I0220 18:45:10.971784 28408 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:45:10.977396 28408 fs_manager.cc:410] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s I0220 18:45:10.981593 28408 fs_manager.cc:430] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0220 18:45:10.982123 28408 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal uuid: "a6e352f77ef54932a47d9977e15d05f1" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:45:10.982231 28408 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0220 18:45:10.989495 27875 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:10.990674 27875 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:10.991171 27893 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:11.017560 28408 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:45:11.018644 28408 ts_tablet_manager.cc:383] Loading tablet metadata (0/1 complete) I0220 18:45:11.021632 28408 ts_tablet_manager.cc:395] Loaded tablet metadata (1 total tablets, 1 live tablets) I0220 18:45:11.021670 28408 ts_tablet_manager.cc:401] Registering tablets (0/1 complete) I0220 18:45:11.022982 28408 ts_tablet_manager.cc:415] Registered 1 tablets I0220 18:45:11.023341 28426 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Bootstrap starting. I0220 18:45:11.035208 28408 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.130:36293 I0220 18:45:11.036168 28408 webserver.cc:382] Webserver started at http://127.23.135.130:39793/ using document root and password file I0220 18:45:11.036716 28408 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/info.pb I0220 18:45:11.039862 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 28408 I0220 18:45:11.039943 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 27727 I0220 18:45:11.045693 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.23.135.131:45349 --local_ip_for_outbound_sockets=127.23.135.131 --webserver_port=35617 --webserver_interface=127.23.135.131 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} W0220 18:45:11.049418 27875 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:11.049531 27882 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:11.050017 27893 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:11.050659 28532 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:45:11.050896 28532 heartbeater.cc:406] Registering TS with master... I0220 18:45:11.051295 28532 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:11.052693 25164 ts_manager.cc:192] Re-registered known tserver with Master: a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293) I0220 18:45:11.053802 25164 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.130:32957 WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:45:11.109427 28535 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:45:11.109632 28535 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:45:11.110926 28535 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:45:11.111037 28535 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.131 I0220 18:45:11.112881 28535 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.131:45349 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/info.pb --webserver_interface=127.23.135.131 --webserver_port=35617 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.28535 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.131 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:45:11.113425 28535 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:45:11.114219 28535 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:45:11.121728 28535 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:45:11.122673 28535 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize W0220 18:45:11.126368 27893 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:11.127506 27893 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:11.131170 28535 hybrid_clock.cc:552] HybridClock initialized: now 1582224311131093 us; error 68 us; skew 500 ppm I0220 18:45:11.131942 28535 fs_manager.cc:259] Metadata directory not provided I0220 18:45:11.131987 28535 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:45:11.133667 28298 log.cc:828] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Log is configured to *not* fsync() on all Append() calls W0220 18:45:11.134045 28543 raft_consensus.cc:659] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: failed to trigger leader election: Illegal state: leader elections are disabled I0220 18:45:11.138927 28535 fs_manager.cc:410] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s I0220 18:45:11.143164 28535 fs_manager.cc:430] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0220 18:45:11.143723 28535 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal uuid: "e4c29ba8de7e4745a4e497d213d63742" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:45:11.143832 28535 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:45:11.171360 28535 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:45:11.172603 28535 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0220 18:45:11.172636 28535 ts_tablet_manager.cc:415] Registered 0 tablets W0220 18:45:11.175837 27893 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39810: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 leader_uuid: "a6e352f77ef54932a47d9977e15d05f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:11.185053 28535 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.131:45349 I0220 18:45:11.185914 28535 webserver.cc:382] Webserver started at http://127.23.135.131:35617/ using document root and password file I0220 18:45:11.186480 28535 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/info.pb I0220 18:45:11.194952 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 28535 I0220 18:45:11.195057 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 27853 W0220 18:45:11.202234 25721 connection.cc:658] client connection to 127.23.135.132:33963 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) I0220 18:45:11.202909 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data --block_manager=log --rpc_bind_addresses=127.23.135.132:33963 --local_ip_for_outbound_sockets=127.23.135.132 --webserver_port=44317 --webserver_interface=127.23.135.132 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} I0220 18:45:11.209123 28659 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:45:11.209286 28659 heartbeater.cc:406] Registering TS with master... I0220 18:45:11.209718 28659 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:11.210708 25164 ts_manager.cc:192] Re-registered known tserver with Master: e4c29ba8de7e4745a4e497d213d63742 (127.23.135.131:45349) I0220 18:45:11.211616 25164 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.131:43173 WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:45:11.298164 28662 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:45:11.298326 28662 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:45:11.299244 28662 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:45:11.299320 28662 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.132 I0220 18:45:11.304764 28662 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.132:33963 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/info.pb --webserver_interface=127.23.135.132 --webserver_port=44317 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.28662 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.132 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:45:11.305266 28662 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:45:11.305919 28662 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:45:11.314788 28662 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:45:11.315531 28662 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:45:11.326966 28426 log.cc:828] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Log is configured to *not* fsync() on all Append() calls I0220 18:45:11.330983 28662 hybrid_clock.cc:552] HybridClock initialized: now 1582224311330935 us; error 44 us; skew 500 ppm I0220 18:45:11.331689 28662 fs_manager.cc:259] Metadata directory not provided I0220 18:45:11.331732 28662 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:45:11.340550 28662 fs_manager.cc:410] Time spent opening directory manager: real 0.008s user 0.006s sys 0.000s I0220 18:45:11.344650 28662 fs_manager.cc:430] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0220 18:45:11.345216 28662 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:45:11.345322 28662 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:45:11.365252 28662 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:45:11.366250 28662 ts_tablet_manager.cc:383] Loading tablet metadata (0/1 complete) I0220 18:45:11.368979 28662 ts_tablet_manager.cc:395] Loaded tablet metadata (1 total tablets, 1 live tablets) I0220 18:45:11.369010 28662 ts_tablet_manager.cc:401] Registering tablets (0/1 complete) I0220 18:45:11.370296 28662 ts_tablet_manager.cc:415] Registered 1 tablets I0220 18:45:11.371979 28680 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Bootstrap starting. I0220 18:45:11.383366 28662 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.132:33963 I0220 18:45:11.384367 28662 webserver.cc:382] Webserver started at http://127.23.135.132:44317/ using document root and password file I0220 18:45:11.384927 28662 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/info.pb I0220 18:45:11.392246 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 28662 I0220 18:45:11.408320 28785 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:45:11.408462 28785 heartbeater.cc:406] Registering TS with master... I0220 18:45:11.408851 28785 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:11.410148 25164 ts_manager.cc:192] Re-registered known tserver with Master: 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) I0220 18:45:11.411162 25164 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.132:47791 I0220 18:45:11.848639 28403 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:45:11.854275 28680 log.cc:828] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Log is configured to *not* fsync() on all Append() calls I0220 18:45:12.055568 28532 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:45:12.213413 28659 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:45:12.412865 28785 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:45:16.141255 28298 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Bootstrap replaying log segment 1/1 (3.58M/4.69M this segment, stats: ops{read=2080 overwritten=0 applied=2080 ignored=0} inserts{seen=103750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0) I0220 18:45:16.328981 28426 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Bootstrap replaying log segment 1/1 (3.69M/4.81M this segment, stats: ops{read=2150 overwritten=0 applied=2147 ignored=0} inserts{seen=107100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0) I0220 18:45:16.721557 28680 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2797 overwritten=0 applied=2797 ignored=0} inserts{seen=139550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0220 18:45:16.721961 28680 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Bootstrap complete. I0220 18:45:16.723274 28680 ts_tablet_manager.cc:1113] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Time spent bootstrapping tablet: real 5.351s user 5.039s sys 0.123s I0220 18:45:16.725870 28680 raft_consensus.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 10 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:16.725963 28680 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c751174fe9e41a0a4d3ea1dac04a50e, State: Initialized, Role: FOLLOWER I0220 18:45:16.726380 28680 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2797, Last appended: 10.2797, Last appended by leader: 2797, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:16.726974 28680 ts_tablet_manager.cc:1142] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s W0220 18:45:16.801793 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:16.859288 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:16.916986 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:17.020457 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:17.071678 28822 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:45:17.071875 28822 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:17.072926 28822 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:45:17.079203 28478 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 11 candidate_status { last_received { term: 10 index: 2797 } } ignore_live_leader: false dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" is_pre_election: true W0220 18:45:17.079766 28673 leader_election.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 11 pre-election: Tablet error from VoteRequest() call to peer a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293): Illegal state: must be running to vote when last-logged opid is not known I0220 18:45:17.084651 28346 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 11 candidate_status { last_received { term: 10 index: 2797 } } ignore_live_leader: false dest_uuid: "853707498adf425fb8395cb60f41c9dc" is_pre_election: true W0220 18:45:17.085120 28671 leader_election.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 11 pre-election: Tablet error from VoteRequest() call to peer 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551): Illegal state: must be running to vote when last-logged opid is not known I0220 18:45:17.085204 28671 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c751174fe9e41a0a4d3ea1dac04a50e; no voters: 853707498adf425fb8395cb60f41c9dc, a6e352f77ef54932a47d9977e15d05f1 I0220 18:45:17.085512 28822 raft_consensus.cc:2675] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority W0220 18:45:17.189712 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:17.248626 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:17.360841 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:17.440641 28822 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:45:17.440791 28822 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:17.441166 28822 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:45:17.441596 28478 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 11 candidate_status { last_received { term: 10 index: 2797 } } ignore_live_leader: false dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" is_pre_election: true I0220 18:45:17.441704 28346 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 11 candidate_status { last_received { term: 10 index: 2797 } } ignore_live_leader: false dest_uuid: "853707498adf425fb8395cb60f41c9dc" is_pre_election: true W0220 18:45:17.441972 28673 leader_election.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 11 pre-election: Tablet error from VoteRequest() call to peer a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293): Illegal state: must be running to vote when last-logged opid is not known W0220 18:45:17.442044 28671 leader_election.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 11 pre-election: Tablet error from VoteRequest() call to peer 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551): Illegal state: must be running to vote when last-logged opid is not known I0220 18:45:17.442091 28671 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c751174fe9e41a0a4d3ea1dac04a50e; no voters: 853707498adf425fb8395cb60f41c9dc, a6e352f77ef54932a47d9977e15d05f1 I0220 18:45:17.442265 28822 raft_consensus.cc:2675] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority I0220 18:45:17.492179 28298 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2748 overwritten=2 applied=2744 ignored=0} inserts{seen=136900 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates I0220 18:45:17.492619 28298 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Bootstrap complete. I0220 18:45:17.493898 28298 ts_tablet_manager.cc:1113] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Time spent bootstrapping tablet: real 6.698s user 6.349s sys 0.127s I0220 18:45:17.496142 28298 raft_consensus.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 10 FOLLOWER]: Replica starting. Triggering 2 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:17.497221 28298 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853707498adf425fb8395cb60f41c9dc, State: Initialized, Role: FOLLOWER I0220 18:45:17.497686 28298 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2744, Last appended: 10.2746, Last appended by leader: 2746, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:17.498253 28298 ts_tablet_manager.cc:1142] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s W0220 18:45:17.531599 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:17.590392 28426 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2797 overwritten=0 applied=2796 ignored=0} inserts{seen=139500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0220 18:45:17.590852 28426 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Bootstrap complete. W0220 18:45:17.590893 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:17.592211 28426 ts_tablet_manager.cc:1113] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Time spent bootstrapping tablet: real 6.569s user 6.359s sys 0.132s I0220 18:45:17.594180 28426 raft_consensus.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 10 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:17.595186 28426 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e352f77ef54932a47d9977e15d05f1, State: Initialized, Role: FOLLOWER I0220 18:45:17.595549 28426 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2796, Last appended: 10.2797, Last appended by leader: 2797, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:17.596042 28426 ts_tablet_manager.cc:1142] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s W0220 18:45:17.649266 28448 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:17.702262 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:17.704880 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:17.764065 28860 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:45:17.764228 28860 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:17.765115 28860 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) W0220 18:45:17.768723 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:17.769937 28742 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 11 candidate_status { last_received { term: 10 index: 2746 } } ignore_live_leader: false dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" is_pre_election: true I0220 18:45:17.770097 28742 raft_consensus.cc:2356] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 853707498adf425fb8395cb60f41c9dc for term 11 because replica has last-logged OpId of term: 10 index: 2797, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 2746. I0220 18:45:17.770119 28478 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 11 candidate_status { last_received { term: 10 index: 2746 } } ignore_live_leader: false dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" is_pre_election: true I0220 18:45:17.770229 28478 raft_consensus.cc:2356] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 853707498adf425fb8395cb60f41c9dc for term 11 because replica has last-logged OpId of term: 10 index: 2797, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 2746. I0220 18:45:17.770586 28291 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 853707498adf425fb8395cb60f41c9dc; no voters: 8c751174fe9e41a0a4d3ea1dac04a50e, a6e352f77ef54932a47d9977e15d05f1 I0220 18:45:17.770854 28860 raft_consensus.cc:2675] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority W0220 18:45:17.817384 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:17.820827 28448 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:17.822743 28822 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:45:17.822858 28822 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:17.823299 28822 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:45:17.823722 28478 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 11 candidate_status { last_received { term: 10 index: 2797 } } ignore_live_leader: false dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" is_pre_election: true I0220 18:45:17.823722 28346 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 11 candidate_status { last_received { term: 10 index: 2797 } } ignore_live_leader: false dest_uuid: "853707498adf425fb8395cb60f41c9dc" is_pre_election: true I0220 18:45:17.823849 28346 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c751174fe9e41a0a4d3ea1dac04a50e in term 10. I0220 18:45:17.823873 28478 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c751174fe9e41a0a4d3ea1dac04a50e in term 10. I0220 18:45:17.824164 28671 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 853707498adf425fb8395cb60f41c9dc, 8c751174fe9e41a0a4d3ea1dac04a50e; no voters: I0220 18:45:17.824317 28822 raft_consensus.cc:2730] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 10 FOLLOWER]: Leader pre-election won for term 11 I0220 18:45:17.824354 28822 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0220 18:45:17.824385 28822 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 10 FOLLOWER]: Advancing to term 11 I0220 18:45:17.825676 28822 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:17.825947 28822 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 11 election: Requested vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:45:17.826248 28478 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 11 candidate_status { last_received { term: 10 index: 2797 } } ignore_live_leader: false dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" I0220 18:45:17.826267 28346 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 11 candidate_status { last_received { term: 10 index: 2797 } } ignore_live_leader: false dest_uuid: "853707498adf425fb8395cb60f41c9dc" I0220 18:45:17.826308 28478 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 10 FOLLOWER]: Advancing to term 11 I0220 18:45:17.826325 28346 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 10 FOLLOWER]: Advancing to term 11 I0220 18:45:17.827373 28346 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c751174fe9e41a0a4d3ea1dac04a50e in term 11. I0220 18:45:17.827386 28478 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c751174fe9e41a0a4d3ea1dac04a50e in term 11. I0220 18:45:17.827670 28671 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 853707498adf425fb8395cb60f41c9dc, 8c751174fe9e41a0a4d3ea1dac04a50e; no voters: I0220 18:45:17.827827 28822 raft_consensus.cc:2730] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 11 FOLLOWER]: Leader election won for term 11 I0220 18:45:17.828016 28822 raft_consensus.cc:686] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 11 LEADER]: Becoming Leader. State: Replica: 8c751174fe9e41a0a4d3ea1dac04a50e, State: Running, Role: LEADER I0220 18:45:17.828312 28822 consensus_queue.cc:229] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2797, Committed index: 2797, Last appended: 10.2797, Last appended by leader: 2797, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:17.830596 25164 catalog_manager.cc:4173] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e reported cstate change: term changed from 10 to 11, leader changed from a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130) to 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132). New cstate: current_term: 11 leader_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } health_report { overall_health: UNKNOWN } } } I0220 18:45:17.889829 28478 raft_consensus.cc:1210] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 11 FOLLOWER]: Refusing update from remote peer 8c751174fe9e41a0a4d3ea1dac04a50e: Log matching property violated. Preceding OpId in replica: term: 10 index: 2797. Preceding OpId from leader: term: 11 index: 2799. (index mismatch) I0220 18:45:17.889834 28346 raft_consensus.cc:1210] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 11 FOLLOWER]: Refusing update from remote peer 8c751174fe9e41a0a4d3ea1dac04a50e: Log matching property violated. Preceding OpId in replica: term: 10 index: 2746. Preceding OpId from leader: term: 11 index: 2799. (index mismatch) I0220 18:45:17.890457 28867 consensus_queue.cc:1005] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [LEADER]: Connected to new peer: Peer: permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2798, Last known committed idx: 2744, Time since last communication: 0.000s I0220 18:45:17.890604 28822 consensus_queue.cc:1005] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2798, Last known committed idx: 2796, Time since last communication: 0.000s I0220 18:45:17.898135 28874 mvcc.cc:203] Tried to move safe_time back from 6480790806070923264 to 6480790805826400256. Current Snapshot: MvccSnapshot[committed={T|T < 6480790806070923264}] I0220 18:45:17.898277 28861 mvcc.cc:203] Tried to move safe_time back from 6480790806070923264 to 6480790805826400256. Current Snapshot: MvccSnapshot[committed={T|T < 6480790806070923264}] I0220 18:45:18.015298 28859 mvcc.cc:203] Tried to move safe_time back from 6480790806070923264 to 6480790805826400256. Current Snapshot: MvccSnapshot[committed={T|T < 6480790806070923264 or (T in {6480790806070923264})}] I0220 18:45:22.587352 25163 ts_manager.cc:267] Unset tserver state for a6e352f77ef54932a47d9977e15d05f1 from MAINTENANCE_MODE I0220 18:45:22.597295 25164 ts_manager.cc:267] Unset tserver state for 8c751174fe9e41a0a4d3ea1dac04a50e from MAINTENANCE_MODE I0220 18:45:22.788985 25164 ts_manager.cc:267] Unset tserver state for 853707498adf425fb8395cb60f41c9dc from MAINTENANCE_MODE I0220 18:45:22.805694 25164 ts_manager.cc:267] Unset tserver state for e4c29ba8de7e4745a4e497d213d63742 from MAINTENANCE_MODE I0220 18:45:22.915289 28532 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:22.967670 28403 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:22.971014 28785 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:23.232726 28659 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:23.366418 25164 ts_manager.cc:278] Set tserver state for 8c751174fe9e41a0a4d3ea1dac04a50e to MAINTENANCE_MODE I0220 18:45:23.403857 25164 ts_manager.cc:278] Set tserver state for 853707498adf425fb8395cb60f41c9dc to MAINTENANCE_MODE I0220 18:45:23.405640 25164 ts_manager.cc:278] Set tserver state for e4c29ba8de7e4745a4e497d213d63742 to MAINTENANCE_MODE I0220 18:45:23.421952 25164 ts_manager.cc:278] Set tserver state for a6e352f77ef54932a47d9977e15d05f1 to MAINTENANCE_MODE I0220 18:45:23.532464 28715 tablet_service.cc:1022] Tablet server 8c751174fe9e41a0a4d3ea1dac04a50e set to quiescing I0220 18:45:23.532580 28715 tablet_service.cc:1029] Tablet server has 1 leaders and 1 scanners I0220 18:45:23.535928 28867 raft_consensus.cc:929] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: : Instructing follower a6e352f77ef54932a47d9977e15d05f1 to start an election I0220 18:45:23.536077 28867 raft_consensus.cc:1017] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 11 LEADER]: Signalling peer a6e352f77ef54932a47d9977e15d05f1 to start an election I0220 18:45:23.541741 28903 raft_consensus.cc:929] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: : Instructing follower a6e352f77ef54932a47d9977e15d05f1 to start an election I0220 18:45:23.541816 28903 raft_consensus.cc:1017] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 11 LEADER]: Signalling peer a6e352f77ef54932a47d9977e15d05f1 to start an election I0220 18:45:23.541890 28903 raft_consensus.cc:929] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: : Instructing follower 853707498adf425fb8395cb60f41c9dc to start an election I0220 18:45:23.541935 28903 raft_consensus.cc:1017] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 11 LEADER]: Signalling peer 853707498adf425fb8395cb60f41c9dc to start an election I0220 18:45:23.542544 28476 tablet_service.cc:1457] Received Run Leader Election RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" from {username='slave'} at 127.23.135.132:45417 I0220 18:45:23.542544 28478 tablet_service.cc:1457] Received Run Leader Election RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" from {username='slave'} at 127.23.135.132:45417 I0220 18:45:23.542634 28476 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 11 FOLLOWER]: Starting forced leader election (received explicit request) I0220 18:45:23.542661 28476 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 11 FOLLOWER]: Advancing to term 12 I0220 18:45:23.543566 28476 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 12 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:23.544086 28822 raft_consensus.cc:929] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: : Instructing follower 853707498adf425fb8395cb60f41c9dc to start an election I0220 18:45:23.544153 28822 raft_consensus.cc:1017] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 11 LEADER]: Signalling peer 853707498adf425fb8395cb60f41c9dc to start an election I0220 18:45:23.544412 28478 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 12 FOLLOWER]: Starting forced leader election (received explicit request) I0220 18:45:23.544572 28478 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 12 FOLLOWER]: Advancing to term 13 I0220 18:45:23.545212 28346 tablet_service.cc:1457] Received Run Leader Election RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" dest_uuid: "853707498adf425fb8395cb60f41c9dc" from {username='slave'} at 127.23.135.132:37903 I0220 18:45:23.545275 28346 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 11 FOLLOWER]: Starting forced leader election (received explicit request) I0220 18:45:23.545300 28346 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 11 FOLLOWER]: Advancing to term 12 I0220 18:45:23.545675 28478 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 13 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:23.546025 28478 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 13 election: Requested vote from peers 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963), 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:45:23.546195 28346 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 12 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:23.546416 28476 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 12 election: Requested vote from peers 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963), 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:45:23.546469 28346 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 12 election: Requested vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) I0220 18:45:23.547780 28354 tablet_service.cc:1457] Received Run Leader Election RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" dest_uuid: "853707498adf425fb8395cb60f41c9dc" from {username='slave'} at 127.23.135.132:37903 I0220 18:45:23.547829 28354 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 12 FOLLOWER]: Starting forced leader election (received explicit request) I0220 18:45:23.547855 28354 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 12 FOLLOWER]: Advancing to term 13 I0220 18:45:23.548125 28742 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 12 candidate_status { last_received { term: 11 index: 3721 } } ignore_live_leader: true dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" I0220 18:45:23.548523 28476 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 12 candidate_status { last_received { term: 11 index: 3721 } } ignore_live_leader: true dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" I0220 18:45:23.548575 28476 raft_consensus.cc:2314] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 853707498adf425fb8395cb60f41c9dc for earlier term 12. Current term is 13. I0220 18:45:23.548632 28354 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 13 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:23.548852 28354 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 13 election: Requested vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) I0220 18:45:23.549206 28742 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 13 candidate_status { last_received { term: 11 index: 3721 } } ignore_live_leader: true dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" I0220 18:45:23.549532 28291 leader_election.cc:403] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 12 election: Vote denied by peer a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293) with higher term. Message: Invalid argument: T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 853707498adf425fb8395cb60f41c9dc for earlier term 12. Current term is 13. I0220 18:45:23.549549 28291 leader_election.cc:406] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term I0220 18:45:23.550024 28476 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "853707498adf425fb8395cb60f41c9dc" candidate_term: 13 candidate_status { last_received { term: 11 index: 3721 } } ignore_live_leader: true dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" I0220 18:45:23.550102 28476 raft_consensus.cc:2339] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 853707498adf425fb8395cb60f41c9dc in current term 13: Already voted for candidate a6e352f77ef54932a47d9977e15d05f1 in this term. I0220 18:45:23.550420 28478 raft_consensus.cc:1175] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 13 FOLLOWER]: Rejecting Update request from peer 8c751174fe9e41a0a4d3ea1dac04a50e for earlier term 11. Current term is 13. Ops: [] I0220 18:45:23.550781 28291 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 853707498adf425fb8395cb60f41c9dc; no voters: 8c751174fe9e41a0a4d3ea1dac04a50e, a6e352f77ef54932a47d9977e15d05f1 I0220 18:45:23.550850 28867 consensus_queue.cc:1016] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 }, Status: INVALID_TERM, Last received: 11.3721, Next index: 3722, Last known committed idx: 3721, Time since last communication: 0.000s I0220 18:45:23.550957 28867 raft_consensus.cc:2978] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 11 LEADER]: Stepping down as leader of term 11 I0220 18:45:23.550974 28867 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 8c751174fe9e41a0a4d3ea1dac04a50e, State: Running, Role: LEADER I0220 18:45:23.551100 28867 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3721, Committed index: 3721, Last appended: 11.3722, Last appended by leader: 3722, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:23.551846 28354 raft_consensus.cc:1175] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 13 FOLLOWER]: Rejecting Update request from peer 8c751174fe9e41a0a4d3ea1dac04a50e for earlier term 11. Current term is 13. Ops: [] I0220 18:45:23.552327 28995 consensus_queue.cc:1016] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 }, Status: INVALID_TERM, Last received: 11.3721, Next index: 3722, Last known committed idx: 3721, Time since last communication: 0.000s I0220 18:45:23.553297 28867 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 11 FOLLOWER]: Advancing to term 13 W0220 18:45:23.553957 28874 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Cannot assign timestamp to transaction. Tablet is not in leader mode. Last heard from a leader: 0.007s ago. W0220 18:45:23.554903 28874 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Cannot assign timestamp to transaction. Tablet is not in leader mode. Last heard from a leader: 0.008s ago. I0220 18:45:23.555770 29056 raft_consensus.cc:2675] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293) with higher term. Message: Invalid argument: T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 853707498adf425fb8395cb60f41c9dc for earlier term 12. Current term is 13. I0220 18:45:23.555864 29057 raft_consensus.cc:2675] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority W0220 18:45:23.557128 28436 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.559855 28436 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:23.560971 28742 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "a6e352f77ef54932a47d9977e15d05f1" candidate_term: 13 candidate_status { last_received { term: 11 index: 3721 } } ignore_live_leader: true dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" I0220 18:45:23.561043 28742 raft_consensus.cc:2356] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a6e352f77ef54932a47d9977e15d05f1 for term 13 because replica has last-logged OpId of term: 11 index: 3722, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 3721. I0220 18:45:23.561648 28742 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "a6e352f77ef54932a47d9977e15d05f1" candidate_term: 12 candidate_status { last_received { term: 11 index: 3721 } } ignore_live_leader: true dest_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" I0220 18:45:23.561684 28354 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "a6e352f77ef54932a47d9977e15d05f1" candidate_term: 13 candidate_status { last_received { term: 11 index: 3721 } } ignore_live_leader: true dest_uuid: "853707498adf425fb8395cb60f41c9dc" I0220 18:45:23.561715 28742 raft_consensus.cc:2314] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a6e352f77ef54932a47d9977e15d05f1 for earlier term 12. Current term is 13. I0220 18:45:23.561770 28354 raft_consensus.cc:2339] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a6e352f77ef54932a47d9977e15d05f1 in current term 13: Already voted for candidate 853707498adf425fb8395cb60f41c9dc in this term. I0220 18:45:23.561972 28346 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "a6e352f77ef54932a47d9977e15d05f1" candidate_term: 12 candidate_status { last_received { term: 11 index: 3721 } } ignore_live_leader: true dest_uuid: "853707498adf425fb8395cb60f41c9dc" I0220 18:45:23.562039 28346 raft_consensus.cc:2314] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a6e352f77ef54932a47d9977e15d05f1 for earlier term 12. Current term is 13. I0220 18:45:23.562206 28417 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a6e352f77ef54932a47d9977e15d05f1; no voters: 853707498adf425fb8395cb60f41c9dc, 8c751174fe9e41a0a4d3ea1dac04a50e I0220 18:45:23.562633 28417 leader_election.cc:403] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 12 election: Vote denied by peer 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) with higher term. Message: Invalid argument: T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a6e352f77ef54932a47d9977e15d05f1 for earlier term 12. Current term is 13. W0220 18:45:23.562640 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:23.562650 28417 leader_election.cc:406] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term I0220 18:45:23.562932 28417 leader_election.cc:403] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [CANDIDATE]: Term 12 election: Vote denied by peer 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) with higher term. Message: Invalid argument: T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a6e352f77ef54932a47d9977e15d05f1 for earlier term 12. Current term is 13. I0220 18:45:23.563550 29058 raft_consensus.cc:2675] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority I0220 18:45:23.563608 29058 raft_consensus.cc:2675] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) with higher term. Message: Invalid argument: T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a6e352f77ef54932a47d9977e15d05f1 for earlier term 12. Current term is 13. W0220 18:45:23.567414 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.572417 28700 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.575965 28700 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.579751 28448 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.585062 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.589258 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.594463 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.597996 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.605022 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.608438 28447 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.614251 28447 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.621299 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.623692 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.636056 28700 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.636505 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.647799 28448 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.650853 28447 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.664232 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.665073 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:23.682194 28330 tablet_service.cc:1022] Tablet server 853707498adf425fb8395cb60f41c9dc set to quiescing W0220 18:45:23.682247 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:23.682307 28330 tablet_service.cc:1029] Tablet server has 0 leaders and 0 scanners W0220 18:45:23.682504 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.697494 28447 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.700260 28448 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.715040 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.719595 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:23.719779 28457 tablet_service.cc:1022] Tablet server a6e352f77ef54932a47d9977e15d05f1 set to quiescing I0220 18:45:23.719951 28457 tablet_service.cc:1029] Tablet server has 0 leaders and 2 scanners I0220 18:45:23.725215 28588 tablet_service.cc:1022] Tablet server e4c29ba8de7e4745a4e497d213d63742 set to quiescing I0220 18:45:23.725338 28588 tablet_service.cc:1029] Tablet server has 0 leaders and 0 scanners W0220 18:45:23.736963 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.740871 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.754571 28447 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.761848 28447 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.775189 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.784324 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.800540 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.805449 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.823932 28448 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.827071 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.844383 28867 raft_consensus.cc:659] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: failed to trigger leader election: Illegal state: leader elections are disabled W0220 18:45:23.846607 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.849726 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.870040 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.874706 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.896575 28448 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.899332 28447 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.923015 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.924731 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.950678 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.954025 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.981160 28448 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:23.982471 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.012233 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.013136 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.043817 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.045781 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.073246 28448 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.078476 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.106678 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.112135 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.143355 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.147899 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.177098 28448 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.181473 28448 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.195698 29089 raft_consensus.cc:659] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: failed to trigger leader election: Illegal state: leader elections are disabled W0220 18:45:24.212532 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.218214 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.237573 29090 raft_consensus.cc:659] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: failed to trigger leader election: Illegal state: leader elections are disabled W0220 18:45:24.248968 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.254509 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.285521 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.293031 28447 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.326277 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.331413 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.369302 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.374402 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.407958 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.413048 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.450655 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.456277 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.495251 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.501821 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.540192 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.545795 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.585918 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.590692 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.634910 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.638494 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:24.655663 28715 tablet_service.cc:1022] Tablet server 8c751174fe9e41a0a4d3ea1dac04a50e set to quiescing I0220 18:45:24.655764 28715 tablet_service.cc:1029] Tablet server has 0 leaders and 1 scanners W0220 18:45:24.680771 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.683899 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.730031 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.733983 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.779163 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.784495 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:24.814198 28457 tablet_service.cc:1022] Tablet server a6e352f77ef54932a47d9977e15d05f1 set to quiescing I0220 18:45:24.814308 28457 tablet_service.cc:1029] Tablet server has 0 leaders and 0 scanners W0220 18:45:24.829154 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.833343 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.881189 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.885154 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.936043 28702 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.936121 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.987898 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:24.989745 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.042423 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.043268 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.098239 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.098829 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.153671 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.154594 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.211388 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.212332 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.269243 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.272363 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.326010 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.333087 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.388043 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.392038 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.449872 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.451730 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.514741 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.515583 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.576503 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.578433 28320 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38088: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.643576 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.644253 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.708231 28447 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:25.708323 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:25.747727 28715 tablet_service.cc:1022] Tablet server 8c751174fe9e41a0a4d3ea1dac04a50e set to quiescing I0220 18:45:25.747814 28715 tablet_service.cc:1029] Tablet server has 0 leaders and 0 scanners I0220 18:45:25.753494 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 28280 I0220 18:45:25.758671 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.23.135.129:34551 --local_ip_for_outbound_sockets=127.23.135.129 --webserver_port=33607 --webserver_interface=127.23.135.129 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} W0220 18:45:25.773664 25721 meta_cache.cc:234] Tablet 7e19d7c23cc941e3a7751fe2478481ec: Replica 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) has failed: Network error: Client connection negotiation failed: client connection to 127.23.135.129:34551: connect: Connection refused (error 111) W0220 18:45:25.776480 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:45:25.803071 29124 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:45:25.803194 29124 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:45:25.804062 29124 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:45:25.804112 29124 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.129 I0220 18:45:25.805382 29124 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.129:34551 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/info.pb --webserver_interface=127.23.135.129 --webserver_port=33607 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.29124 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.129 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:45:25.805784 29124 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:45:25.806329 29124 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:45:25.811050 29124 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:45:25.811779 29124 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:45:25.812901 29124 hybrid_clock.cc:552] HybridClock initialized: now 1582224325812882 us; error 26 us; skew 500 ppm I0220 18:45:25.813344 29124 fs_manager.cc:259] Metadata directory not provided I0220 18:45:25.813374 29124 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:45:25.817898 29124 fs_manager.cc:410] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s I0220 18:45:25.820777 29124 fs_manager.cc:430] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0220 18:45:25.821171 29124 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal uuid: "853707498adf425fb8395cb60f41c9dc" format_stamp: "Formatted at 2020-02-20 18:44:40 on dist-test-slave-dist-test-slave-sbf6" I0220 18:45:25.821270 29124 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:45:25.836966 29124 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:45:25.837747 29124 ts_tablet_manager.cc:383] Loading tablet metadata (0/1 complete) I0220 18:45:25.840524 29124 ts_tablet_manager.cc:395] Loaded tablet metadata (1 total tablets, 1 live tablets) I0220 18:45:25.840560 29124 ts_tablet_manager.cc:401] Registering tablets (0/1 complete) I0220 18:45:25.841831 29124 ts_tablet_manager.cc:415] Registered 1 tablets I0220 18:45:25.843369 29142 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Bootstrap starting. W0220 18:45:25.845911 28449 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34830: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:25.848635 29124 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.129:34551 I0220 18:45:25.849392 29124 webserver.cc:382] Webserver started at http://127.23.135.129:33607/ using document root and password file I0220 18:45:25.849800 29124 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-0/data/info.pb I0220 18:45:25.852507 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 29124 I0220 18:45:25.852591 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 28408 I0220 18:45:25.859910 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.23.135.130:36293 --local_ip_for_outbound_sockets=127.23.135.130 --webserver_port=39793 --webserver_interface=127.23.135.130 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} I0220 18:45:25.860276 29247 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:45:25.860393 29247 heartbeater.cc:406] Registering TS with master... I0220 18:45:25.860731 29247 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:25.861889 25164 ts_manager.cc:192] Re-registered known tserver with Master: 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:45:25.862677 25164 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.129:49021 W0220 18:45:25.908521 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:25.920164 25751 meta_cache.cc:1038] Marking tablet server a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293) as failed. WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:45:25.947248 29251 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:45:25.947391 29251 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:45:25.952363 29251 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:45:25.952440 29251 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.130 I0220 18:45:25.954025 29251 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.130:36293 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/info.pb --webserver_interface=127.23.135.130 --webserver_port=39793 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.29251 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.130 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:45:25.954517 29251 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:45:25.955147 29251 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:45:25.959993 29251 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:45:25.960695 29251 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:45:25.961863 29251 hybrid_clock.cc:552] HybridClock initialized: now 1582224325961835 us; error 24 us; skew 500 ppm I0220 18:45:25.962361 29251 fs_manager.cc:259] Metadata directory not provided I0220 18:45:25.962397 29251 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:45:25.967211 29251 fs_manager.cc:410] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s I0220 18:45:25.970324 29251 fs_manager.cc:430] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s I0220 18:45:25.970736 29251 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal uuid: "a6e352f77ef54932a47d9977e15d05f1" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:45:25.970824 29251 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0220 18:45:25.987491 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:26.016508 25750 meta_cache.cc:1038] Marking tablet server a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293) as failed. I0220 18:45:26.033165 29251 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:45:26.034005 29251 ts_tablet_manager.cc:383] Loading tablet metadata (0/1 complete) I0220 18:45:26.036303 29251 ts_tablet_manager.cc:395] Loaded tablet metadata (1 total tablets, 1 live tablets) I0220 18:45:26.036336 29251 ts_tablet_manager.cc:401] Registering tablets (0/1 complete) I0220 18:45:26.037586 29251 ts_tablet_manager.cc:415] Registered 1 tablets I0220 18:45:26.043453 29251 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.130:36293 I0220 18:45:26.043952 29269 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Bootstrap starting. I0220 18:45:26.045982 29251 webserver.cc:382] Webserver started at http://127.23.135.130:39793/ using document root and password file I0220 18:45:26.046512 29251 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-1/data/info.pb W0220 18:45:26.046847 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:26.047152 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 29251 I0220 18:45:26.047216 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 28535 I0220 18:45:26.053959 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.23.135.131:45349 --local_ip_for_outbound_sockets=127.23.135.131 --webserver_port=35617 --webserver_interface=127.23.135.131 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} I0220 18:45:26.065227 29374 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:45:26.065351 29374 heartbeater.cc:406] Registering TS with master... I0220 18:45:26.065743 29374 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:26.067149 25164 ts_manager.cc:192] Re-registered known tserver with Master: a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293) I0220 18:45:26.068614 25164 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.130:43355 WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:45:26.120416 29377 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:45:26.120626 29377 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:45:26.121901 29377 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:45:26.122041 29377 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.131 I0220 18:45:26.124063 29377 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.131:45349 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/info.pb --webserver_interface=127.23.135.131 --webserver_port=35617 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.29377 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.131 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:45:26.125104 29377 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:45:26.125973 29377 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:45:26.132717 29377 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:45:26.133648 29377 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:45:26.141000 29377 hybrid_clock.cc:552] HybridClock initialized: now 1582224326140960 us; error 43 us; skew 500 ppm I0220 18:45:26.141659 29377 fs_manager.cc:259] Metadata directory not provided I0220 18:45:26.141703 29377 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:45:26.148419 29377 fs_manager.cc:410] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s I0220 18:45:26.152688 29377 fs_manager.cc:430] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0220 18:45:26.153323 29377 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal uuid: "e4c29ba8de7e4745a4e497d213d63742" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:45:26.153424 29377 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:45:26.203033 29377 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:45:26.204140 29377 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0220 18:45:26.204177 29377 ts_tablet_manager.cc:415] Registered 0 tablets I0220 18:45:26.207113 29142 log.cc:828] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Log is configured to *not* fsync() on all Append() calls I0220 18:45:26.212483 29377 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.131:45349 I0220 18:45:26.213344 29377 webserver.cc:382] Webserver started at http://127.23.135.131:35617/ using document root and password file I0220 18:45:26.213891 29377 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-2/data/info.pb W0220 18:45:26.219137 28701 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:39908: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:26.220755 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 29377 I0220 18:45:26.220846 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 28662 I0220 18:45:26.232781 24094 external_mini_cluster.cc:1065] Running /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data --block_manager=log --rpc_bind_addresses=127.23.135.132:33963 --local_ip_for_outbound_sockets=127.23.135.132 --webserver_port=44317 --webserver_interface=127.23.135.132 --tserver_master_addrs=127.23.135.190:41437 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.135.148:46155 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --scanner_default_batch_size_bytes=100 with env {} I0220 18:45:26.234840 29500 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:45:26.235000 29500 heartbeater.cc:406] Registering TS with master... I0220 18:45:26.235430 29500 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:26.236400 25164 ts_manager.cc:192] Re-registered known tserver with Master: e4c29ba8de7e4745a4e497d213d63742 (127.23.135.131:45349) I0220 18:45:26.237269 25164 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.131:59631 WARNING: Logging before InitGoogleLogging() is written to STDERR W0220 18:45:26.305066 29503 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0220 18:45:26.305240 29503 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0220 18:45:26.306162 29503 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=1024 W0220 18:45:26.306228 29503 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.135.132 I0220 18:45:26.307613 29503 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.135.148:46155 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_heartbeat_interval_ms=100 --fs_data_dirs=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data --fs_wal_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.135.132:33963 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/info.pb --webserver_interface=127.23.135.132 --webserver_port=44317 --tserver_master_addrs=127.23.135.190:41437 --scanner_default_batch_size_bytes=100 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.29503 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.135.132 --log_dir=/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 924cf36232aa121073c75b5863683510c05eae48 build type DEBUG built by None at 20 Feb 2020 18:30:41 UTC on jenkins-slave-8qw8 build id 20512 I0220 18:45:26.308204 29503 env_posix.cc:1682] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0220 18:45:26.308921 29503 file_cache.cc:476] Constructed file cache file cache with capacity 419430 I0220 18:45:26.313980 29503 hybrid_clock.cc:241] auto-selected time source: builtin I0220 18:45:26.314690 29503 hybrid_clock.cc:534] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0220 18:45:26.315812 29503 hybrid_clock.cc:552] HybridClock initialized: now 1582224326315793 us; error 17 us; skew 500 ppm I0220 18:45:26.316313 29503 fs_manager.cc:259] Metadata directory not provided I0220 18:45:26.316352 29503 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0220 18:45:26.320966 29503 fs_manager.cc:410] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s I0220 18:45:26.324296 29503 fs_manager.cc:430] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0220 18:45:26.324702 29503 fs_manager.cc:447] Opened local filesystem: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data,/tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" format_stamp: "Formatted at 2020-02-20 18:44:41 on dist-test-slave-dist-test-slave-sbf6" I0220 18:45:26.324796 29503 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal metadata directory: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/wal 1 data directories: /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0220 18:45:26.362018 29503 env_posix.cc:1682] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0220 18:45:26.362954 29503 ts_tablet_manager.cc:383] Loading tablet metadata (0/1 complete) I0220 18:45:26.365259 29503 ts_tablet_manager.cc:395] Loaded tablet metadata (1 total tablets, 1 live tablets) I0220 18:45:26.365293 29503 ts_tablet_manager.cc:401] Registering tablets (0/1 complete) I0220 18:45:26.366302 29503 ts_tablet_manager.cc:415] Registered 1 tablets I0220 18:45:26.368896 29520 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Bootstrap starting. I0220 18:45:26.377188 29503 rpc_server.cc:211] RPC server started. Bound to: 127.23.135.132:33963 I0220 18:45:26.379180 29503 webserver.cc:382] Webserver started at http://127.23.135.132:44317/ using document root and password file I0220 18:45:26.379778 29503 server_base.cc:678] Dumped server information to /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094/minicluster-data/ts-3/data/info.pb I0220 18:45:26.389475 24094 external_mini_cluster.cc:1127] Started /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver as pid 29503 I0220 18:45:26.404150 29625 heartbeater.cc:322] Connected to a master server at 127.23.135.190:41437 I0220 18:45:26.404289 29625 heartbeater.cc:406] Registering TS with master... I0220 18:45:26.404680 29625 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:26.405877 25164 ts_manager.cc:192] Re-registered known tserver with Master: 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) I0220 18:45:26.406919 25164 master_service.cc:350] Signed X509 certificate for tserver {username='slave'} at 127.23.135.132:49033 I0220 18:45:26.510123 29269 log.cc:828] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Log is configured to *not* fsync() on all Append() calls I0220 18:45:26.731386 29520 log.cc:828] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Log is configured to *not* fsync() on all Append() calls I0220 18:45:26.863986 29247 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:45:27.070528 29374 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:45:27.239148 29500 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:45:27.408591 29625 heartbeater.cc:447] Master 127.23.135.190:41437 was elected leader, sending a full tablet report... I0220 18:45:31.208515 29142 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Bootstrap replaying log segment 1/1 (3.97M/6.36M this segment, stats: ops{read=2305 overwritten=2 applied=2300 ignored=0} inserts{seen=114700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0) I0220 18:45:31.513854 29269 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Bootstrap replaying log segment 1/1 (4.11M/6.41M this segment, stats: ops{read=2385 overwritten=0 applied=2385 ignored=0} inserts{seen=118950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0) I0220 18:45:31.737412 29520 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Bootstrap replaying log segment 1/1 (3.87M/6.41M this segment, stats: ops{read=2246 overwritten=0 applied=2246 ignored=0} inserts{seen=112050 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0) I0220 18:45:34.307973 29520 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3722 overwritten=0 applied=3721 ignored=0} inserts{seen=185700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0220 18:45:34.308387 29520 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Bootstrap complete. I0220 18:45:34.309876 29520 ts_tablet_manager.cc:1113] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Time spent bootstrapping tablet: real 7.942s user 7.674s sys 0.156s I0220 18:45:34.312350 29520 raft_consensus.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 13 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:34.313320 29520 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 13 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c751174fe9e41a0a4d3ea1dac04a50e, State: Initialized, Role: FOLLOWER I0220 18:45:34.313711 29520 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3721, Last appended: 11.3722, Last appended by leader: 3722, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:34.315672 29520 ts_tablet_manager.cc:1142] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s W0220 18:45:34.332171 29536 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:40004: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:34.491183 29536 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:40004: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:34.648021 29698 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:45:34.648211 29698 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:34.649140 29698 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:45:34.654600 29320 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 14 candidate_status { last_received { term: 11 index: 3722 } } ignore_live_leader: false dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" is_pre_election: true W0220 18:45:34.655257 29513 leader_election.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 14 pre-election: Tablet error from VoteRequest() call to peer a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293): Illegal state: must be running to vote when last-logged opid is not known I0220 18:45:34.656987 29195 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 14 candidate_status { last_received { term: 11 index: 3722 } } ignore_live_leader: false dest_uuid: "853707498adf425fb8395cb60f41c9dc" is_pre_election: true W0220 18:45:34.657395 29511 leader_election.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 14 pre-election: Tablet error from VoteRequest() call to peer 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551): Illegal state: must be running to vote when last-logged opid is not known I0220 18:45:34.657464 29511 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c751174fe9e41a0a4d3ea1dac04a50e; no voters: 853707498adf425fb8395cb60f41c9dc, a6e352f77ef54932a47d9977e15d05f1 I0220 18:45:34.657701 29698 raft_consensus.cc:2675] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority W0220 18:45:34.776008 29536 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:40004: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:34.863157 29142 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=3723 overwritten=2 applied=3721 ignored=0} inserts{seen=185700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0220 18:45:34.863598 29142 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Bootstrap complete. I0220 18:45:34.865499 29142 ts_tablet_manager.cc:1113] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Time spent bootstrapping tablet: real 9.022s user 8.627s sys 0.204s I0220 18:45:34.868242 29142 raft_consensus.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 13 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:34.868322 29142 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 13 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853707498adf425fb8395cb60f41c9dc, State: Initialized, Role: FOLLOWER I0220 18:45:34.868770 29142 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3721, Last appended: 11.3721, Last appended by leader: 3721, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:34.869360 29142 ts_tablet_manager.cc:1142] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s W0220 18:45:34.938758 29536 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:40004: Illegal state: Replica 8c751174fe9e41a0a4d3ea1dac04a50e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:34.983602 29269 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=3721 overwritten=0 applied=3721 ignored=0} inserts{seen=185700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0220 18:45:34.983999 29269 tablet_bootstrap.cc:441] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Bootstrap complete. I0220 18:45:34.985401 29269 ts_tablet_manager.cc:1113] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Time spent bootstrapping tablet: real 8.942s user 8.551s sys 0.168s I0220 18:45:34.987493 29269 raft_consensus.cc:346] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 13 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:34.987556 29269 raft_consensus.cc:726] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 13 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e352f77ef54932a47d9977e15d05f1, State: Initialized, Role: FOLLOWER I0220 18:45:34.987953 29269 consensus_queue.cc:252] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3721, Last appended: 11.3721, Last appended by leader: 3721, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:34.988436 29269 ts_tablet_manager.cc:1142] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s I0220 18:45:35.064321 29698 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0220 18:45:35.064465 29698 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:35.064800 29698 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:45:35.065317 29320 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 14 candidate_status { last_received { term: 11 index: 3722 } } ignore_live_leader: false dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" is_pre_election: true I0220 18:45:35.065320 29195 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 14 candidate_status { last_received { term: 11 index: 3722 } } ignore_live_leader: false dest_uuid: "853707498adf425fb8395cb60f41c9dc" is_pre_election: true I0220 18:45:35.065491 29320 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c751174fe9e41a0a4d3ea1dac04a50e in term 13. I0220 18:45:35.065498 29195 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c751174fe9e41a0a4d3ea1dac04a50e in term 13. I0220 18:45:35.065858 29513 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c751174fe9e41a0a4d3ea1dac04a50e, a6e352f77ef54932a47d9977e15d05f1; no voters: I0220 18:45:35.066036 29698 raft_consensus.cc:2730] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 13 FOLLOWER]: Leader pre-election won for term 14 I0220 18:45:35.066068 29698 raft_consensus.cc:482] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0220 18:45:35.066082 29698 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 13 FOLLOWER]: Advancing to term 14 I0220 18:45:35.067353 29698 raft_consensus.cc:504] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:35.067610 29698 leader_election.cc:293] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 14 election: Requested vote from peers a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293), 853707498adf425fb8395cb60f41c9dc (127.23.135.129:34551) I0220 18:45:35.067987 29320 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 14 candidate_status { last_received { term: 11 index: 3722 } } ignore_live_leader: false dest_uuid: "a6e352f77ef54932a47d9977e15d05f1" I0220 18:45:35.068001 29195 tablet_service.cc:1340] Received RequestConsensusVote() RPC: tablet_id: "7e19d7c23cc941e3a7751fe2478481ec" candidate_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" candidate_term: 14 candidate_status { last_received { term: 11 index: 3722 } } ignore_live_leader: false dest_uuid: "853707498adf425fb8395cb60f41c9dc" I0220 18:45:35.068055 29320 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 13 FOLLOWER]: Advancing to term 14 I0220 18:45:35.068061 29195 raft_consensus.cc:2983] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 13 FOLLOWER]: Advancing to term 14 I0220 18:45:35.069092 29320 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c751174fe9e41a0a4d3ea1dac04a50e in term 14. I0220 18:45:35.069164 29195 raft_consensus.cc:2414] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c751174fe9e41a0a4d3ea1dac04a50e in term 14. I0220 18:45:35.069383 29513 leader_election.cc:307] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c751174fe9e41a0a4d3ea1dac04a50e, a6e352f77ef54932a47d9977e15d05f1; no voters: I0220 18:45:35.069545 29698 raft_consensus.cc:2730] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 14 FOLLOWER]: Leader election won for term 14 I0220 18:45:35.069700 29698 raft_consensus.cc:686] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [term 14 LEADER]: Becoming Leader. State: Replica: 8c751174fe9e41a0a4d3ea1dac04a50e, State: Running, Role: LEADER I0220 18:45:35.070008 29698 consensus_queue.cc:229] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3721, Committed index: 3721, Last appended: 11.3722, Last appended by leader: 3722, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } I0220 18:45:35.072213 25164 catalog_manager.cc:4173] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e reported cstate change: term changed from 11 to 14. New cstate: current_term: 14 leader_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } health_report { overall_health: UNKNOWN } } } W0220 18:45:35.076028 29157 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38202: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 14 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:35.088416 29157 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38202: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 14 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:35.148166 29320 raft_consensus.cc:1210] T 7e19d7c23cc941e3a7751fe2478481ec P a6e352f77ef54932a47d9977e15d05f1 [term 14 FOLLOWER]: Refusing update from remote peer 8c751174fe9e41a0a4d3ea1dac04a50e: Log matching property violated. Preceding OpId in replica: term: 11 index: 3721. Preceding OpId from leader: term: 14 index: 3723. (index mismatch) I0220 18:45:35.148836 29698 consensus_queue.cc:1005] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3723, Last known committed idx: 3721, Time since last communication: 0.000s I0220 18:45:35.154553 29195 raft_consensus.cc:1210] T 7e19d7c23cc941e3a7751fe2478481ec P 853707498adf425fb8395cb60f41c9dc [term 14 FOLLOWER]: Refusing update from remote peer 8c751174fe9e41a0a4d3ea1dac04a50e: Log matching property violated. Preceding OpId in replica: term: 11 index: 3721. Preceding OpId from leader: term: 14 index: 3723. (index mismatch) I0220 18:45:35.155069 29698 consensus_queue.cc:1005] T 7e19d7c23cc941e3a7751fe2478481ec P 8c751174fe9e41a0a4d3ea1dac04a50e [LEADER]: Connected to new peer: Peer: permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3723, Last known committed idx: 3721, Time since last communication: 0.000s W0220 18:45:35.160133 29157 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:38202: Illegal state: Replica 853707498adf425fb8395cb60f41c9dc is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 14 leader_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:35.166615 29287 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34926: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 14 leader_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:35.237893 29287 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34926: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 14 leader_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } W0220 18:45:35.240263 29287 tablet_service.cc:670] failed transaction from {username='slave'} at 127.0.0.1:34926: Illegal state: Replica a6e352f77ef54932a47d9977e15d05f1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 14 leader_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e352f77ef54932a47d9977e15d05f1" member_type: VOTER last_known_addr { host: "127.23.135.130" port: 36293 } } peers { permanent_uuid: "8c751174fe9e41a0a4d3ea1dac04a50e" member_type: VOTER last_known_addr { host: "127.23.135.132" port: 33963 } } peers { permanent_uuid: "853707498adf425fb8395cb60f41c9dc" member_type: VOTER last_known_addr { host: "127.23.135.129" port: 34551 } } } I0220 18:45:35.547785 25752 scanner-internal.cc:113] Scan of tablet 7e19d7c23cc941e3a7751fe2478481ec at a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293) deadline exceeded after 10.000s I0220 18:45:35.932373 25751 scanner-internal.cc:113] Scan of tablet 7e19d7c23cc941e3a7751fe2478481ec at 8c751174fe9e41a0a4d3ea1dac04a50e (127.23.135.132:33963) deadline exceeded after 10.000s I0220 18:45:36.032660 25750 scanner-internal.cc:113] Scan of tablet 7e19d7c23cc941e3a7751fe2478481ec at a6e352f77ef54932a47d9977e15d05f1 (127.23.135.130:36293) deadline exceeded after 10.000s I0220 18:45:37.411746 25168 ts_manager.cc:267] Unset tserver state for a6e352f77ef54932a47d9977e15d05f1 from MAINTENANCE_MODE I0220 18:45:37.598286 25168 ts_manager.cc:267] Unset tserver state for 8c751174fe9e41a0a4d3ea1dac04a50e from MAINTENANCE_MODE I0220 18:45:37.621444 25168 ts_manager.cc:267] Unset tserver state for e4c29ba8de7e4745a4e497d213d63742 from MAINTENANCE_MODE I0220 18:45:37.622869 25165 ts_manager.cc:267] Unset tserver state for 853707498adf425fb8395cb60f41c9dc from MAINTENANCE_MODE /home/jenkins-slave/workspace/kudu-master/2/src/kudu/integration-tests/maintenance_mode-itest.cc:750: Failure Value of: s.ok() Actual: true Expected: false /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/test_util.cc:345: Failure Failed Timed out waiting for assertion to pass. I0220 18:45:37.644279 24094 external_mini_cluster-itest-base.cc:80] Found fatal failure I0220 18:45:37.644357 24094 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 0 with UUID 853707498adf425fb8395cb60f41c9dc and pid 29124 ************************ BEGIN STACKS ************************** 0x00007f94cd2cfd4d in ?? () Id Target Id Frame * 1 process 29124 "kudu-tserver" 0x00007f94cd2cfd4d in ?? () Thread 1 (process 29124): #0 0x00007f94cd2cfd4d in ?? () #1 0x0000000000000000 in ?? () ************************* END STACKS *************************** I0220 18:45:37.839855 24094 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 1 with UUID a6e352f77ef54932a47d9977e15d05f1 and pid 29251 ************************ BEGIN STACKS ************************** 0x00007f523477ed4d in ?? () Id Target Id Frame * 1 process 29251 "kudu-tserver" 0x00007f523477ed4d in ?? () Thread 1 (process 29251): #0 0x00007f523477ed4d in ?? () #1 0x0000000000000000 in ?? () ************************* END STACKS *************************** I0220 18:45:38.028971 24094 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 2 with UUID e4c29ba8de7e4745a4e497d213d63742 and pid 29377 ************************ BEGIN STACKS ************************** I0220 18:45:38.160323 29374 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:38.164572 29247 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... I0220 18:45:38.180331 29625 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... 0x00007f79e8a0ad4d in ?? () Id Target Id Frame * 1 process 29377 "kudu-tserver" 0x00007f79e8a0ad4d in ?? () Thread 1 (process 29377): #0 0x00007f79e8a0ad4d in ?? () #1 0x0000000000000000 in ?? () ************************* END STACKS *************************** I0220 18:45:38.229574 24094 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 3 with UUID 8c751174fe9e41a0a4d3ea1dac04a50e and pid 29503 I0220 18:45:38.258072 29500 heartbeater.cc:455] Master 127.23.135.190:41437 requested a full tablet report, sending... ************************ BEGIN STACKS ************************** 0x00007f6df282ad4d in ?? () Id Target Id Frame * 1 process 29503 "kudu-tserver" 0x00007f6df282ad4d in ?? () Thread 1 (process 29503): #0 0x00007f6df282ad4d in ?? () #1 0x0000000000000000 in ?? () ************************* END STACKS *************************** I0220 18:45:38.418167 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 29124 I0220 18:45:38.423765 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 29251 I0220 18:45:38.428553 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 29377 I0220 18:45:38.431833 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-tserver with pid 29503 I0220 18:45:38.436558 24094 external_mini_cluster.cc:1263] Killing /tmp/dist-test-taskDlYUX2/build/debug/bin/kudu-master with pid 25142 2020-02-20T18:45:38Z chronyd exiting I0220 18:45:38.451180 24094 test_util.cc:143] ----------------------------------------------- I0220 18:45:38.451200 24094 test_util.cc:144] Had fatal failures, leaving test files at /tmp/dist-test-taskDlYUX2/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1582224264261468-24094 [ FAILED ] RollingRestartArgs/RollingRestartITest.TestWorkloads/4, where GetParam() = 264-byte object <01-00 00-00 04-00 00-00 D8-13 75-4C 09-7F 00-00 D8-13 75-4C 09-7F 00-00 00-00 00-00 00-00 00-00 D8-13 75-4C 09-7F 00-00 01-00 00-00 00-00 00-00 00-5D 4D-02 00-00 00-00 10-5D 4D-02 00-00 00-00 ... 00-00 00-00 00-00 00-00 30-18 4F-02 00-00 00-00 30-18 4F-02 00-00 00-00 30-18 4F-02 00-00 00-00 01-00 00-00 00-00 00-00 01-00 00-00 00-00 00-00 04-00 00-00 03-00 00-00 01-62 4D-02 00-00 00-00> (57750 ms) [----------] 1 test from RollingRestartArgs/RollingRestartITest (57751 ms total) [----------] Global test environment tear-down [==========] 2 tests from 2 test cases ran. (74185 ms total) [ PASSED ] 1 test. [ FAILED ] 1 test, listed below: [ FAILED ] RollingRestartArgs/RollingRestartITest.TestWorkloads/4, where GetParam() = 264-byte object <01-00 00-00 04-00 00-00 D8-13 75-4C 09-7F 00-00 D8-13 75-4C 09-7F 00-00 00-00 00-00 00-00 00-00 D8-13 75-4C 09-7F 00-00 01-00 00-00 00-00 00-00 00-5D 4D-02 00-00 00-00 10-5D 4D-02 00-00 00-00 ... 00-00 00-00 00-00 00-00 30-18 4F-02 00-00 00-00 30-18 4F-02 00-00 00-00 30-18 4F-02 00-00 00-00 01-00 00-00 00-00 00-00 01-00 00-00 00-00 00-00 04-00 00-00 03-00 00-00 01-62 4D-02 00-00 00-00> 1 FAILED TEST